The number was right. What it counted was wrong.
A review record from our own internal council, 22 September 2026. This is how we check our own work before it ships. It is not the MegaLens product pipeline, though it runs on the same idea.
We were planning a content programme, and one statistic was going to carry it.
In April we ran security audits on 32 small public projects on GitHub. One AI model, which we will call Model A, was asked to add up the results. It came back with this:
362 findings across 32 projects. 39% rated critical. 58% of them found at the audit's second stage rather than its first.
That last number looked like the best argument we could make for a second opinion. The plan put it in the first article, the research page, and a press release.
It was wrong. Not invented. Every count was real and correctly added up. What was wrong was the assumption about what was being counted.
What one record actually looked like
The audit files store findings as a list. Model A counted the list. Here is the start of one entry it counted as a single “finding”:
# Security Audit: <project name> ## Critical Findings ### 1. No Authentication/Authorization on Endpoints
That is not a finding. It is the beginning of a whole report, cut off after 120 characters, and stored as one record. It contains several findings nobody ever separated out.
152 of the 362 records were like this. So the total was not 362. And the “critical” label came from the first heading in each report, which is why so many of them were marked critical.
How it was caught
Model A had also checked its own work. The numbers added up, so the check passed. A self-review compares the result against the author's own understanding, and here the understanding was the problem.
We then gave the same data to two other models, Model B and Model C, each from a different AI company, and asked them to attack the plan. The instructions said plainly that “looks fine” would count as a failed review.
Model B
Did not trust the summary. It opened the raw files, recounted, and read the records themselves. It found the report-sized entries.
Model C
Worked separately and got there another way. It noticed the categories lined up exactly with the stages of the audit: every second-stage item was labelled one way, every first-stage item the other. That is what labelling by stage looks like, not evidence of discovery.
Both said do not ship this. Then we opened the files ourselves before acting on either review. Both were right.
What the data really said
| What the first analysis said | What was true |
|---|---|
| 362 findings | 210 separate findings, plus 152 whole reports nobody had split apart |
| 39% critical | 6.7%, 14 of the 210 |
| 58% found by the second stage | Could not be worked out. It divided findings by a mix of findings and reports |
| Every finding confirmed | "Confirmed" was the default value. The broken records were "confirmed" too |
Model B and Model C found two more problems the headline numbers would never have shown. Most of the second-stage findings, 181 of 210, came from a single model inside the audit, so the “several models” story was weaker than it looked. And the 32 projects were mostly starter kits and demos. Nothing showed their code was written by AI, so they could not support a claim about AI-written code at all.
What it would have cost
The review took one request each to Model B and Model C and was finished the same afternoon.
Without it, the first thing we published would have been a research page built on a false number, with a press release attached. A press release goes out to many sites at once, and it cannot be edited once it has gone. A corrected web page is an inconvenience. A wrong number syndicated across news sites is permanent.
What changed
We dropped the statistic. Nothing we publish uses 362, 39% or 58%. We are rerunning the audits on a new sample, with each finding stored on its own, and every project's origin checked in a way anyone can repeat.
The part worth keeping
This was not a hallucination. No fact was made up and no sum was wrong. Model A checked its own work carefully and passed it, because the mistake was in the question it never asked: what is this field actually counting?
A second reviewer who does not share your assumptions will test them. You will not, because to you they do not look like assumptions. Model B and Model C, from two different companies, working independently, both found it.
That is the argument for a second opinion, and this time it was made at our own expense.
We do not name the models. The lesson is about one model checking its own work, not about which model it was.
Method: one analysis by one model, reviewed by two models from two other AI companies, working independently from the same data package, followed by our own check of the raw records. The audited projects are not named and no finding from them is quoted. This record comes from our internal review council, not from a MegaLens product run.