My pipeline ruled a true claim false, then destroyed four correct findings with it

MegaLens runs your code past several AI engines from different families, working independently. An audit stage rules on what they produced, because engines are confident about things that are not true. A third piece, the claim checker, verifies factual premises by web search, so the pipeline does not build a fix on something false.

That last piece is what this post is about. It did the opposite of its job and took four good findings with it.

The record, before I explain it

The run’s stored record, out of my own database. An excerpt: citation titles and some disposition fields are left out for length, nothing else is changed.

claim:      "bcrypt-nodejs is a valid npm package that provides bcrypt hashing functionality"
verdict:    "contradicted"
supports:   "S01, S02"
sourceSays: "Native JS implementation of BCrypt for Node."

sources:
  https://www.npmjs.com/package/bcrypt
  https://www.npmjs.com/package/bcrypt-node
  https://codeforgeek.com/password-hashing-nodejs/
  https://www.npmjs.com/package/bcryptjs
  https://www.npmjs.com/package/bcrypt/v/0.7.0
  https://www.npmjs.com/package/bcrypt/v/3.0.5

partial_approval:  { proposed: 4, validated: 4, rejected: 0 }

And this is what the npm registry says, fetched while I was writing this:

GET registry.npmjs.org/bcrypt-nodejs  ->  200
latest:       0.0.3, published 2013-02-25
description:  "A native JS bcrypt library for NodeJS."
deprecated:   "bcrypt-nodejs is no longer actively maintained. Please use bcrypt or bcryptjs."

Read those two blocks against each other. Most of what follows is naming what you can already see.

Defect A: it never looked at the thing it was judging

Six citations, and not one is bcrypt-nodejs. The verifier answered a question about bcrypt-nodejs using pages about bcrypt, bcrypt-node, bcryptjs and a blog post: different packages by different authors, two of them the ones you are told to migrate to.

Then look at the sentence it offered as the refutation:

Native JS implementation of BCrypt for Node.

And the package’s own description on npm:

A native JS bcrypt library for NodeJS.

It found a paraphrase of the claim and filed it as evidence against the claim.

The rule that let this through is one line, and it is the whole problem:

rawVerdict === 'contradicted' && sources.length > 0

It requires citations. It never requires a citation about the subject. Any six URLs satisfy it. The module’s own header sets a stricter bar in prose, saying a contradiction needs affirmative incompatibility and that silence and ambiguity are unverified. The bar was written in a comment and enforced by a length check.

There was also no stored reasoning. The verifier was never asked which part of the claim its quote denies, so a contradiction could not be audited afterwards even in principle. I caught this one only because it happened to be about a package I could look up.

Defect B: a claim about two steps killed all four findings

This one is independent of the first, and it is more expensive.

Look at the record again: supports: "S01, S02". The checker knew its own scope. The plan had four steps. S01 and S02 were about swapping in bcrypt. S03 was a login error message leaking whether an account exists. S04 was a date function producing days that do not exist.

The decision site reads a boolean over the entire run and stops. The supports field is never consulted there. It is read once, to print the failure message.

So a disputed claim about the remedy in two steps threw away all four findings, including one about passwords stored in plaintext, marked critical, true whether or not any npm package exists.

The part that took me a while to accept: the checker’s real job had already worked. It runs before the proposer so a doubtful premise cannot be built on, and it did exactly that. The proposer produced four findings, the audit passed all four, the per-finding gates passed all four. Then the run died on a second penalty for a claim the first had already handled.

Defect C: “valid” was three claims wearing one coat

bcrypt-nodejs is a valid npm package is not one proposition but three. Does it exist? Yes. Does it do bcrypt hashing? Yes. Is it maintained? No: npm carries an explicit deprecation notice.

The extractor wrote a compound sentence with an evaluative adjective in it, and the verifier was free to answer whichever reading its search results suited. One verdict cannot stand for a claim whose readings disagree. That is not a model failure. It is a question that should never have been asked in that form.

The worst part was none of the above

The same row that says validated: 4, rejected: 0 showed the customer this:

They did NOT pass the audit stage, so treat them as a second opinion to weigh, not as confirmed defects.

Four findings that passed every gate, handed over labelled as having failed. If you were that user, you would have thrown away a real plaintext-password finding because the tool told you to.

I assumed the message was reading some stale field. It is worse. The sentence was a string constant inside the halt branch, emitted whenever a run stopped with findings attached. It did not read the wrong record. It read no record. It could not have been right, because nothing about it was derived from anything.

Where the decision and the sentence describing it come from two different places, the one the customer sees will eventually be the wrong one. A customer-visible statement has to come from the same record the decision was made in, enforced by a test rather than by remembering.

What changed

The checker always returned a named verdict. The yes or no was downstream: the decision site collapsed it into one boolean for the whole run. It now works on a typed outcome: supported, contradicted, source unavailable, ambiguous claim, or unverified.

contradicted has to be earned. At least one citation must resolve to the actual subject. The verifier must state which part of the claim its quote denies, and that statement is stored, so a contradiction can be argued with later. A quote that merely restates the claim counts as evidence for it. Compound and evaluative predicates are split into single checkable propositions first, and readings that would get different answers come out ambiguous rather than false.

For npm, PyPI, crates.io, GitHub repositories and CVE identifiers there is no reason to ask a model at all: a registry request settles existence outright. Deprecation is weaker than it looks, and I had this wrong too. npm’s flag can apply to a single version and its message is free text, so it is not a maintained or unmaintained switch. It gives you the maintainer’s own statement, which here says plainly to use something else. Worth quoting, not worth rounding into a verdict.

A contradiction is applied per finding through its declared scope, not to the whole run. A checker that cannot verify something downgrades what depends on it and never vetoes. “Deprecated” annotates the recommendation, where the problem actually is, and leaves the finding alone. And the caveat is derived from the decision record, with a test that fails if any wording says nothing passed audit while the record shows something did.

How it was found, which is the real point

I did not find this by reading the code. I ran twenty ordinary reviews to decide whether the product was good enough to invite people to. One came back with four findings labelled as having failed audit, which is a different thing from coming back empty, and is why it nearly went unexamined. Then somebody reviewing the results noticed that bcrypt-nodejs does, in fact, exist on npm.

That is the uncomfortable bit. My pipeline is built on the belief that one model checking its own work is not enough, and the check that failed here was a single verifier marking its own homework, with nothing asking whether it had looked at the right page. I had applied the idea to the engines and not to the machinery around them.

The question came from outside. It usually does.

Appendix: the fixtures

These are the cases the fix has to pass. Each one is written as a test rather than a note, so a regression puts them back in front of me.

1. Run 16 exactly as stored: four findings validated, with the recommendation in S01 and S02 annotated as pointing at an unmaintained package.
2. A contradicted premise that touches one finding of four: the other three come back by exact id.
3. An undeclared dependency discovered during the check: that finding is withheld.
4. A relation that cannot be determined: the finding goes to needs verification, not to rejected.
5. A shared premise that genuinely undermines all four: none survive, and the run says so.
6. A package that exists but is deprecated: existence and maintenance come out as separate answers.
7. An evidence fetch that fails: the outcome is unknown, never contradicted.
8. A registry lookup that returns 404: contradicted, correctly. The gates must not become a machine that can never say no.