Failure catalogue · The verification is theater · 26 of 51

OBSERVED FAILURE MODE

The verify script that silently did nothing.

An agent asked to verify its own work writes a verification script, runs it, and reports that everything passed. The script matched no files, or exited before its assertions, or compared an empty set with an empty set. A check that never executed is indistinguishable from a check that found nothing wrong. The output of both is identical.

What we saw

In our own audits, an agent wrote itself a verification pass over a build, ran it, and reported clean. The script did nothing. It examined no rows and reported the absence of failures as success, so the log filled with passing lines that described no work at all. The figures it blessed were on their way to an executive deck. Reading the script did not reveal the problem, because the code looked like verification and the run looked like a green suite.

Why it passes a glance

Green is green. A suite that finds nothing wrong and a suite that examines nothing produce the same summary line, the same exit code, and the same confidence in a reviewer. Nobody re-reads a passing test, and the one signal that separates the two cases, a run that turns red on known-bad input, was never taken.

What addresses it

The verification-practice skill requires every guard to be sabotaged: write the case that fails on bad input as well as the case that passes on good input, then revert the fix, watch the case go red through the deployed path, restore it, and watch it pass. That is Principle 6, test that a check fires, not that it exists. A check that cannot fire is a comment with a runtime cost.

Check your own file in two minutes

1. Feed the check a file you have broken on purpose and confirm it blocks.
2. Read the run log for the count of items actually examined, not the count of assertions passed.
3. Remove the input entirely and confirm the check reports a refusal rather than a pass.
4. Where a check has no recorded red run, stop rendering it as a pass.

What this does not catch

A fire test proves one guard fires on one known defect. It does not prove the guard covers every shape of that defect, and a sabotage that reproduces a neighbouring scenario proves nothing about the original. Coverage across parallel code paths is a separate question with its own answer.

Quick answers

How can a passing test suite be worthless?

When nothing in it can fail. A suite that examines an empty set, or exits before its assertions, reports the same green as a suite that checked everything. The distinguishing evidence is a captured red run on known-bad input.

What is a fire test?

A test that feeds a guard input it should reject, through the production code path, and confirms the guard blocks. It proves the guard is capable of failing, which no passing run can show on its own.

Should verification be written by the agent that built the report?

It can be, provided the guard is sabotaged afterwards and the red run is captured. Authorship is not the risk here. An unexercised guard is the risk, whoever wrote it.

Nearby failures

Last updated 2026-09-02 · Dylan, founder · one of 51 observed failure modes, every one seen in a real build or in our own audits, none invented.