Failure catalogue · The verification is theater · 29 of 51
OBSERVED FAILURE MODE
Fire-test theater.
The fire test exists, is named after the guard, and passes. It also re-implements the guard's comparison inline instead of calling the deployed code, so it exercises a copy that lives only inside the test. The guard can be broken for months while the test that names it stays green. Evidence is claimed; nothing was exercised.
What we saw
In our own audits, a test written to prove a guard fires never called that guard. It reproduced the comparison inline, fed the bad input to its own copy, watched the copy reject it, and reported success. The deployed guard was never invoked. Rewiring or renaming the real check would not have turned the test red. This shape survives every review that asks whether a fire test exists for a check, because one does exist, with the right name, in the right file, passing.
Why it passes a glance
It answers the audit question as asked. A reviewer looking for a fire test per guard finds one per guard. The test's own assertions are correct, its bad input really is rejected, and the rejection really is reported. The missing element, a call into the production path, is invisible in a summary and easy to read past in the source.
What addresses it
The adversarial-review skill rules this out by construction: never test through a re-implementation of the call path, run the real interface, and sabotage the deployed code so the case goes red for the measured defect and green again once restored. Principle 6, test that a check fires, not that it exists, requires the fire test to call the deployed guard through the production code path.
Check your own file in two minutes
- Open each fire test and find the line that calls the deployed guard.
- Rename or disable the real guard, then confirm the test turns red.
- Run the suite from a clean checkout rather than the working tree.
- Treat any test that stays green under a broken guard as evidence of nothing.
What this does not catch
Calling the production path proves the deployed guard ran. It does not prove the sabotage reproduced the defect the guard exists to stop, and a partial sabotage that still passes proves redundancy rather than protection. Only the complete sabotage tells the truth.
Quick answers
- How do I know my fire test really calls the guard?
- Break the guard and run the test. If it stays green, the test is exercising a copy. Renaming the guard is the fastest version of this, since a test that imports the real thing cannot survive the rename.
- Why run tests from a clean checkout?
- Because a test that is green only in the tree it was written in is not evidence. Uncommitted files, local fixtures, and stale artifacts all keep suites passing that would fail for anybody else.
- Is one sabotage per guard enough?
- It is the minimum. The sabotage must reproduce the original measured defect rather than a neighbouring one, and where a rule lives in several code paths, each path needs its own case.
Nearby failures
Last updated 2026-09-02 · Dylan, founder · one of 51 observed failure modes, every one seen in a real build or in our own audits, none invented.