Research, September 2026
LLM speed in real code review: what public benchmarks did not tell us.
We spent about $95 in model fees trying to beat our own lineup of AI models. In the main test we changed one model at a time, and checked each review against bugs we already knew were there. Most swaps lost bugs. Models that look fast on paper could not finish in time. And in the same week, a model we relied on stopped working.
363
full review runs in the main test
9 of 10
known bugs found by the lineup we kept
4 of 10
found by the fastest lineup
0.59 to 0.99
how closely time followed answer length
Security reviews of web apps, September 2026. An AI model matched the results to findings an earlier study had already settled. No human reviewed the results.
The setup: one goal, beat our own lineup
Our review sends your code to AI models from different companies. Each result names the models that took part. We wanted to know if a different lineup would do better.
So we used 11 real codebases we had already studied. An earlier fact check had settled 17 of their findings: 10 real bugs and 7 false alarms. That gave us an answer key.
We ran our current lineup first. Then, in the main test, we swapped in one new model at a time and ran the full review again. Every setup ran 3 times on all 11 codebases. A bug counted as found when at least 2 of the 3 runs found it. We ranked by one rule, in this order: real bugs found, then speed, then cost.
The main test was 363 full review runs. With the follow up runs we did afterwards, it cost about $95 in model fees.
LLM timeouts: response time tracked answer length far more than input length
People do not like to wait, so every answer in our review has a time limit. Several models that look fast in public speed charts could not finish inside it on our real code.
Grok 4.6
Ran out of time in 33 of 33 runs on a task that needs a long written answer. On a shorter task it did very well.
GLM 5.3 Flash
Ran out of time in 32 of 33 runs.
DeepSeek V4 Flash
Ran out of time in 27 of 33 runs.
DeepSeek V4.1 Flash
The fastest DeepSeek model on OpenRouter’s published data: 0.76 s to the first token, 80 tokens a second. In our review it ran out of time in 12 of 33 runs, and the reviews it took part in found 5 of the 10 known bugs.
Claude Sonnet 5 and Grok 4.7
Ran out of time on some runs of one task.
Public first token speeds did not predict full answer times in our tests. Our inputs were 5,000 to 100,000 tokens of code. So we measured it ourselves.
LLM latency comparison: seven models, the same job
Seven models did the same job on the same codebases. For each one we compared the time an answer took with two things: how long the question was, and how long the answer was.
| Model | Typical time | Answer length (tokens) | Writing speed (tokens a second) | Time follows answer length | Time follows question length |
|---|---|---|---|---|---|
| Muse Spark 1.2 | 18 s | 4,205 | 222 | 0.79 | barely (0.08) |
| Gemini 3.1 Pro | 19 s | 2,621 | 133 | 0.99 | no (0.0) |
| Gemini 3.8 Flash | 22 s | 1,996 | 89 | 0.91 | some (0.65) |
| GPT 5.4 | 29 s | 3,058 | 105 | 0.95 | no (below 0) |
| Grok 4.6 | 38 s | 2,498 | 61 | 0.61 | no (below 0) |
| Grok 4.7 | 44 s * | 3,176 | 75 | 0.98 | a little (0.30) |
| Claude Sonnet 5 | 50 s * | 4,471 | 91 | 0.59 | barely (0.07) |
* Runs that finished in time only. The others ran out of time, so the real typical time is higher. The last two columns are correlations: 1 means time rises in step with that length, 0 means no link.
The pattern is clear. For every model, answer length had a stronger link with time than input length. For most models the link with input length was close to zero or below.
Claude Sonnet 5 wrote long answers at a normal speed. The two Grok models wrote more slowly.
Check: if an LLM keeps timing out, look at how much it writes before you cut your prompt. Try a shorter answer and measure whether it helps.
LLM speed comparison: the fastest lineup found the fewest bugs
The fastest lineup in the whole test swapped in a lighter, faster model. It cut a typical review from 91 seconds to 82 seconds, about 9 seconds faster.
It found 4 of the 10 known bugs. The lineup we already had found 9.
It was not alone. Every single swap in the main test found fewer real bugs than our current lineup: between 4 and 8 of 10.
A review that is 9 seconds faster and misses 5 more bugs is not a faster review. It is a worse one. Speed only counts after the bugs are found.
The best bug finder can be the wrong model to check the findings
One model was among the best we tested at finding problems. In a full review it found as many known bugs as our current lineup. It also found a bug our starting lineup missed.
We also gave the same model a different task: to judge whether each finding was real. It confirmed two findings that were false alarms. The best models on that task confirmed none.
Same model, same code, two jobs, two very different results. A score in one job told us almost nothing about the other.
Check: test a model on the exact job you will give it. Finding problems and checking them are different skills.
OpenRouter 404, model not found: a model we used disappeared
On 23 September a model we used stopped working. OpenRouter had removed Devstral, a coding model from Mistral, completely. Calls to it returned 404.
The error message told us to move to a newer model id. That was the id we were already calling. It returned 404 too.
Our last successful call to that model was at 08:23 UTC. The fix went live at 22:21 UTC, almost 14 hours later. Our alerts did not catch the failure. The call ran outside the part of the review we log, and our alerts read that log. A person testing the product found it before our monitoring did.
That night we checked every model id we call against OpenRouter. Five were already dead, and three of those were in use. Two more had published end dates less than a month away.
What we changed
- Every call now reports a withdrawn model the moment it happens, and we get an email.
- A daily check reads each model's published end date and warns us 14 days ahead.
If you build on hosted models
- Log every model call, including the ones outside your main flow.
- Alert on “model not found” on its own. It is not the same as a slow or busy provider.
- Watch the end dates your provider publishes. They give you days of warning for free.
- Do not trust the replacement an error message suggests until you have called it yourself.
Hitting a different OpenRouter error? We wrote up OpenRouter errors and how to fix each one: 404, 429, 401 and “provider returned error”.
How to choose an LLM for code review: what we changed and what we kept
The main test supported keeping most of our lineup: no swap in it beat the lineup on real bugs found. Other tests helped us choose the few changes we made.
So most of the money bought confidence, not change. We now know which models hold up on our real workload, and why the others did not.
What this does not show
- A precise ranking. There were only 10 known bugs, and runs vary: our own lineup found 7, 9 and 7 across its three runs. A gap of one bug is within that noise.
- How these models do on other work. We tested security reviews of web apps, nothing else.
- How they will do next month. Providers change models, and so do the results.
That last point is why we keep checking. Every day we check that each model we use still answers. We watch published end dates and get a warning two weeks ahead. And measurements like this one are how we decide what to change. That habit is the backbone of the product.
Questions people ask
Why does my LLM keep timing out?
Longer answers tended to take more time in our tests. For each model we measured, time had a stronger link with answer length than with input length. So check how much the model writes before you cut your prompt. Try a shorter answer and measure whether it helps.
Does a longer prompt make an LLM slower?
It varied by model in our tests. The link between answer length and time was strong for all seven models we measured, with correlations from 0.59 to 0.99. The link with input length was 0.65 at most, and close to zero or below for five of the seven.
Is the fastest LLM the best one for coding tasks?
Not in our test. The fastest lineup we tried cut about 9 seconds from a typical review and found 4 of 10 known bugs. The lineup we kept found 9 of 10.
What does "OpenRouter 404 model not found" mean?
In our case, OpenRouter had removed the model. The error told us to use the same model id we were already calling. It still returned 404. Check the live model list and test the id before you use it.
How do I choose the best model for code review?
Test candidates on code where you already know the real bugs. Change one model at a time and run each setup more than once. Count real bugs found first, then false alarms, then speed, then cost. Public speed figures did not predict which models would finish in time in our tests.
Method
September 2026, with the same review code customers use, on 11 web app codebases from an earlier study. None is named. The answer key is the 20 findings that study checked: 10 agreed real, 7 agreed not real, 3 unresolved and left out.
In the main test, each alternative setup changed one model. Each setup ran 3 times on every codebase. A bug counted as found when 2 of the 3 runs found it. An AI model matched each review's findings to the answer key without seeing which findings were real. No human reviewed the results. The speed table shows median time per answer. The lineup comparison shows median time per full review. Costs are model fees as reported by OpenRouter.