How do I tell a specific check from a coincidental one?

Ask which literals would change for a different month, region or account. Then move a row between groups in a copy of the input: if a correct build fails, the check encodes trivia about one file rather than a rule.

Should checks never contain values?

They should hold mechanism, not judgment. Tolerances, bounds, alias lists and content markers live in configuration where a person can read and change them, and an empty setting fails loudly rather than passing quietly.

What does this cost the second customer?

An afternoon or a rebuild. That line, code holds the mechanism and configuration holds the judgment, is exactly what decides which of the two the second customer turns out to be.

Failure catalogue · The verification is theater · 41 of 51

OBSERVED FAILURE MODE

The check that proves a coincidence.

A guard written from the literals of a single file, one region, one invoice id, one month, will redden truthfully whenever that file changes and prove nothing about the next export. A check belongs to the property it asserts, never to the dataset it was written beside. Move a row between groups and a correct build fails.

What we saw

In our own audits a guard was built from values that happened to hold for the file in front of it: one region, one invoice id, one month. It had a fire test, it turned red on bad input, and it read as a real gate on the page beside the rest. It was measuring recognition rather than correctness. The next export, legitimately different in region or period, would have failed a correct build, and a wrong number that left those literals intact would have sailed straight past it.

Why it passes a glance

It behaves like a working check on the only file anyone has run. Green when the build is good, red when the input is broken, evidence attached. Literals also read to a reviewer as specificity, which reads as rigour. The defect appears one tenant later, when the check either blocks a correct build or waves a wrong figure through.

What addresses it

The adversarial-review skill draws the line: reproduction confirms presence, not scope, and breadth stays an unproven claim until a sweep asserts the rule everywhere it should hold. Doctrine principle 8, Thresholds are configuration, not code, says where the literals belong. Code holds the mechanism, configuration holds the judgment, and empty configuration fails loudly instead of defaulting to permissive.

Check your own file in two minutes

List every literal in the check and ask which would change for a different month, region, or account.
Move a row between groups in a copy of the input and confirm a correct build still passes.
Move the judgment values into configuration and confirm an empty setting fails loudly.

What this does not catch

Writing to the property widens what a check covers, not what it understands. A guard can be perfectly general and still assert the wrong property. State in one sentence what each guard claims, so the next reader has something to disagree with.

Quick answers

How do I tell a specific check from a coincidental one?: Ask which literals would change for a different month, region or account. Then move a row between groups in a copy of the input: if a correct build fails, the check encodes trivia about one file rather than a rule.
Should checks never contain values?: They should hold mechanism, not judgment. Tolerances, bounds, alias lists and content markers live in configuration where a person can read and change them, and an empty setting fails loudly rather than passing quietly.
What does this cost the second customer?: An afternoon or a rebuild. That line, code holds the mechanism and configuration holds the judgment, is exactly what decides which of the two the second customer turns out to be.

Nearby failures

Fire-test theaterThe verification is theater Borrowed evidenceThe verification is theater The check that cannot failThe verification is theater

Last updated 2026-09-02 · Dylan, founder · one of 51 observed failure modes, every one seen in a real build or in our own audits, none invented.

Get the free skill → The doctrine All 51 failure modes