Research, September 2026
We ran AI code review on 30 apps. Then we fact-checked our own findings.
The review came back with 160 findings. We took a random sample of 20 and had two AI models from different companies check each one against the code, independently. On the findings they agreed about, 44% did not hold up. And not one “critical” label was rated critical by both.
30
apps reviewed
20
findings checked
44%
false positives on agreed findings
0 of 9
critical labels kept by both checkers
Validated by two AI models from different companies. No human reviewed these findings, and no exploit was attempted. A sample of 20 has a wide error bar.
What we ran
30 public web apps from GitHub, in three groups. 25 of them were written with AI, which is the code this study is about.
| Group | Apps | How we identified it |
|---|---|---|
| Built by an AI web builder | 15 | Identified by the builder’s own build plugin in package.json |
| Written by developers with AI help | 10 | Identified by an AI co-author line in the commit history, or a README saying it was built with an AI coding tool |
| An earlier sample, rerun | 5 | Projects from an earlier study, run again on the exact same input |
Every app got the same security review with the same settings: its security-relevant files, up to about 28,000 characters. Each finished run used three or four AI models from different companies. The whole batch took 18 minutes and cost $5.67 in model fees. 28 of the 30 runs finished; 2 stopped part way and are counted as results, not retried.
What came back
160 findings across the 28 finished runs, a median of 5 per app. Every finished run reported at least two.
That is the number a review tool would normally lead with. We did not want to publish it until we had checked how far to trust it.
Then we checked our own findings
We drew 20 of the 160 at random, with a fixed seed, spread across the three groups and weighted toward the findings labelled critical, because those carry the most weight.
Two AI models from different companies each checked all 20 against the actual code. For the 15 findings whose projects still exist, that meant the exact commit that was reviewed, and the rest of the project too. The other 5 came from projects that have since been deleted, so they were checked against the exact code the review saw. The second checker was not shown the first one's answers. The first checker's verdicts were saved and fingerprinted before the second one finished, so neither could be adjusted to match the other.
We only count a verdict where both checkers agreed.
| Outcome | Findings |
|---|---|
| Both checkers agree the finding is real | 9 |
| Both checkers agree it is not real | 7 |
| Unresolved: the checkers disagreed, or one could not tell | 4 |
On the 16 findings both agreed on, 7 were not real: a false-positive rate of about 44%. Put simply: of the findings both checkers agreed on, a little over half held up and a little under half did not.
Severity did worse. Nine of the sampled findings were labelled critical. Not one was rated critical by both checkers. On the nine findings both agreed were real, the second checker rated eight of them lower than the first.
Why findings failed: three causes you can check for
Each of these accounted for more than one false positive. They are worth knowing whatever review tool you use.
1. Thin input
This review reads a slice of a project, not all of it. In one run the slice was a single file of data types, from an app with no backend at all. The models reasoned about a server that did not exist, and two findings came out of that.
Check: Check how much of the project the review actually saw. A finding about code that was never shown to the reviewer is a guess.
2. Protection that lives somewhere else
One finding said an endpoint had no cross-site check. The first function that endpoint calls does exactly that check. Another said a database function did not validate a value; the table itself rejects bad values, and only a trusted role can call the function.
Check: Before accepting "X is missing", follow the call one level down and look at the database constraints.
3. Intended behaviour read as a flaw
Admins seeing revenue on an admin-only page. An analytics counter treated as if it were a login session. A password-reset link built from the page’s own address, which an attacker cannot change.
Check: Ask whether the behaviour is what the app is supposed to do. If it is, it is not a vulnerability.
And the problem with severity
The labels look like they were set by pattern: a secret, a missing check, a missing signature. They do not reflect whether the code is ever called, or what an attacker could actually do with it. Code that nothing in the app calls was labelled critical twice.
Check: before acting on a critical label, confirm the code is reachable and that the app is deployed the way the finding assumes.
Where the two checkers disagreed
Four findings stayed unresolved. They are left out of the 44%, but they are the most useful part of the exercise, because each one is a real question a review has to answer, and a single checker would have answered it without knowing there was a question.
An analytics beacon sent without authentication
One checker: real, but low risk, because a browser beacon cannot hold a secret. The other: cannot tell without seeing the script that receives it.
An analytics sheet readable from the browser
One checker: real, low value data. The other: cannot tell, because whether the sheet is public is a setting, not code.
A form that uses a random link as the only key
One checker: a real design limit, low severity. The other: intended design for an anonymous form, not a flaw.
A developer tool that falls back from a container to the host
One checker: real, medium. The other: the container mode refuses the fallback; it only happens in the automatic mode, where running on the host is the intended behaviour.
In the last case we went back to the code afterwards: the second checker had read the file that decides the fallback, and the first had not. We still count it as unresolved, because the rule was set before the answers came in.
The pattern across all four: the disagreement is exactly where the context lives. Who calls this code, what the setting is, what the app was built to do. A review that hides disagreement hides the questions you most need to ask.
Did more models agreeing help?
The obvious guess is that a finding raised by several models is more trustworthy. On the findings both checkers agreed about, it was not.
| Finding raised by | Real | Not real |
|---|---|---|
| One model | 4 | 1 |
| Two or more models | 5 | 5 |
| No model recorded | 0 | 1 |
Findings raised by several models were real 5 times out of 10. Findings raised by a single model were real 4 times out of 5. The numbers are small, so we would not call it a pattern yet. But they do not support the easy story.
What they do support is narrower: a second model surfaces what the first one missed. It does not make what they agree on true. That is why MegaLens shows where its models disagreed, instead of handing back a single agreed answer.
What this does not show
- How many real vulnerabilities these apps have. The sample says how far to trust a review, not how safe the apps are.
- That code from one group is better or worse than another. The groups are too small to compare.
- A precise false-positive rate. Twenty findings give a direction, with a wide error bar.
- What a human expert would conclude. No human reviewed these findings.
Questions people ask
How accurate is AI code review?
In our sample, less accurate than its output suggests. Two AI models from different companies checked 20 findings independently. On the 16 they agreed about, 9 were real and 7 were not, a false-positive rate of about 44%. It is a sample of 20, so read it as a warning, not a precise rate.
What causes false positives in AI code review?
Three causes each accounted for more than one false positive: the reviewer saw too little of the project, the protection existed in a function or database rule it did not look at, and intended behaviour was read as a flaw.
Can you trust the severity labels from AI code review?
Not on their own. Nine findings in our sample were labelled critical. Neither checker kept every one of them at critical, and not one was rated critical by both. The labels look set by pattern, without weighing whether the code is reachable or what an attacker could actually do with it.
Does using more than one AI model make findings more reliable?
Not in the way people assume. Findings raised by several models were real 5 times out of 10 in our sample. Findings raised by a single model were real 4 times out of 5. A second model surfaces what the first missed. It does not make what they agree on true.
Method
Reviews were run on 22 September 2026 with MegaLens, a security review on the same settings for every app, on a dedicated test account. The apps are public GitHub projects whose owners did not ask to be reviewed, so none is named and no finding is quoted. Nobody was contacted.
Sample: 20 findings, drawn at random with a fixed seed, stratified by group and weighted toward the critical label. Verification: two AI models from different companies, working independently from the same code and the same findings. Each classed every finding as confirmed, real but overstated, or not real; the second checker could also answer cannot tell. The published rate uses only findings where both agreed. Validated by AI models only. No human review, no exploit attempted, no live system touched.