Failure catalogue · The verification is theater · 40 of 51

OBSERVED FAILURE MODE

The check that reads its own configuration.

A verdict built from a configuration flag reports what the build was told to do, not what the build did. A check that consults its own settings is a mirror, not a gate: it cannot see the render step drop the marker the setting promised. Fire test, evidence file and clean gate run sit above it, all of them true.

What we saw

In our own audits a check had everything a reader looks for: a genuine fire test through the production path, its own evidence file naming that exact check, and a clean gate run around it. Its verdict was nonetheless computed from a flag the same build had set to true. The flag said the marker would render. The render step dropped it, the page showed the affected figure clean, and the check row still reported that disclosure had happened, because it had never looked at the page.

Why it passes a glance

Every visible quality signal is present, which is what lets this shape survive review. The fire test is real, because flipping the flag does turn the row red. The evidence file is unique. What no artifact shows is the distance between the setting and the shipped page, and a reader auditing check rows sees a verdict, never the expression that produced it.

What addresses it

The verification-practice skill says to recompute from the artifact's own bytes, never to import a helper from the code under audit, and never to test through a re-implementation of the call path. Doctrine principle 6, Test that a check fires, not that it exists, applies per rendered row: the row reads the shipped artifact, and a row whose fire test proves only that a flag toggles may not render as PASS.

Check your own file in two minutes

1. Read the line that computes each verdict and name the input it actually reads.
2. Reject any verdict whose input is a setting, a flag, or a variable the same build assigned.
3. Delete the marker from the rendered page by hand, rerun the check, and confirm it turns red.

What this does not catch

Reading the artifact proves the marker is present. It does not prove the marker sits on the right figure or carries the right stakes, and it says nothing about the rollups that figure feeds. Placement, disclosure content and every aggregate need checks of their own.

Quick answers

The check has a fire test. Is that not enough?

The fire test proves the row can turn red. If the input it reads is a setting the same build assigned, all it proves is that the setting toggles. The row still cannot see what the render step shipped to the reader.

What should a check read instead?

The shipped artifact: the page, the export, the file that leaves the building. Recompute from those bytes, and never import a helper from the code being checked in order to check it.

Are configuration flags ever a legitimate input?

For deciding what to do, yes. For deciding what happened, no. A verdict is a statement about an artifact, so its input is that artifact, read after the build finished writing it.

Nearby failures

Last updated 2026-09-02 · Dylan, founder · one of 51 observed failure modes, every one seen in a real build or in our own audits, none invented.