THE FAILURE CATALOGUE
Why raw AI outputs aren't boardroom ready.
An agent will build you a dashboard in an afternoon. The dashboard will be confident everywhere, and wrong in ways that pass every glance. Here are fifty-one of those ways, catalogued, and what closes each one.
The number is plausible, and wrong
The dangerous class. Nothing here fails an arithmetic check; every figure reconciles, sums, and renders. It is simply not true.
The 100x slip that ties perfectly. A price entered as 5900.00 instead of 59.00 multiplies through every subtotal consistently, so every reconciliation passes while the headline triples.
✓Principle 11: reconciliation proves arithmetic, not plausibility. A mandatory screen compares every value against its group's typical, both directions, and flags the row with its share of every headline it feeds.
The quantity slip the price screen misses. Screens tuned to prices sail straight past a 100x seats or units error, which certifies exactly as cleanly.
✓Principle 11 names quantity, seat, unit, and count columns explicitly. We know because our own audits proved a seats slip shipped where the wording primed a price-only reading.
The total that repeats on every line. CRM and invoice exports carry the deal total on each line item. Sum the obvious column and revenue lands at a multiple of the truth; we measured one real export summing 9x high.
✓The hostile-exports skill names the total-in-a-column trap and requires the naive sum, the word wrong, and the grain-aware figure printed side by side.
The Total row that is also data. A file ships its own summary row; the pipeline sums it along with the rows it summarizes and doubles the answer.
✓Anchor discovery treats embedded totals as anchors to tie against, never rows to sum.
The lift a holiday window tripled. A promotion analysis headlined a lift near three times its true size because the comparison window quietly absorbed a holiday.
✓Period discipline separates the comparison window from the reporting period, and flags carry a computed what-if under each open reading.
The trend that was one purchase order. A product looked unresponsive to promotion until the entire signal traced to a single large order; another claimed 6x volume that was 1.8x.
✓Flags carry stakes: every flagged row states its share of each headline it feeds, so one row wearing a whole trend is visible on the page.
Phantom margin from one bad cost line. A single corrupted cost value manufactured margin swings across an entire category.
✓The plausibility screen flags the line, and the what-if branch reprices it by code so the corrected figure is on the page, not left as homework.
The wrong column that reconciles. Read the wrong column with the right shape and every total still ties. The number is internally perfect and means the wrong thing.
✓Principle 2: two independent reads that share no logic, down to the parse. One method cannot fool two.
The file lies
Vendor exports are named, encoded, and structured by the system that produced them, not for the person about to trust them.
The filename that lies about the period. A file called Q2 carrying July rows. Headline the file and the quarter is wrong.
✓Principle 4: contents decide the period, and out-of-period rows get a reconciliation bridge, never a blend.
The date that parses two valid ways. 4/9/2026 is April or September depending on a locale the file never states, and both parses succeed.
✓Principle 3: prove the format from rows the file itself disambiguates, disclose the proof, and stop if nothing settles it.
Two date formats in one column. ISO dates and US dates interleaved in the same export, parsing cleanly into silently mixed months.
✓The same format proof runs per row, not per file; a column that changes dialect mid-stream is an error, not a parse.
The encoding that crashes the pipeline. A non-UTF-8 export that kills the reader mid-run, or worse, half-parses.
✓The landing pass archives raw bytes first and proves the encoding before any value is read.
Names doing the work of keys. No stable identifier, so two accounts with the same name silently merge and their history becomes fiction.
✓Principle 3: a label that appears more than once is an error to raise, never a join to guess.
The substring that misfiles a customer. Channel classification by name matching, until a person's surname contains a retailer's name and an account lands in the wrong channel.
✓Classification is relabelling, so it goes through a reviewable override table held as configuration, with the human verdict recorded.
Products living only in free text. The product a row concerns exists nowhere but a prose field, and the extraction rules grow into dozens of patterns nobody audits.
✓Principle 8: extraction terms are configuration with coverage reported, not regexes buried in code.
The join that quietly covers 62%. A reference join drops a third of rows and the report says nothing, because nothing asked.
✓Join coverage is a published number with a threshold; below it, the build fails closed instead of averaging the survivors.
The platform grading its own homework. Ad platforms report the conversions they believe they caused; summed, they routinely claim more revenue than the business made.
✓Principle 9: platform numbers are claims, labelled with claimant and window, and never blended into reconciled figures.
The metric means something else
Every definition below is internally consistent, reconciles perfectly, and passes every gate. The failure is meaning, which arithmetic cannot see.
Two churn definitions, 3.4 points apart. Two teams, one word, different denominators, both defensible, and a leadership deck that mixes them.
✓Principle 10: resolve the fork before computing, record the ruling, and show the definition beside the number.
Sell-through's four denominators. Units received, beginning on-hand, sold plus ending, average inventory: four numbers, one name, and a marketplace variant that exceeds 100%.
✓The Metric Library catalogues the forks and the exact question each needs answered before the figure exists.
Gross, net, and total from one admin. A single storefront backend emits three revenues before accounting adds theirs. Which one is on the slide?
✓The definition renders beside the number, every time, so the slide answers the question itself.
Lift as a multiple or a percent. One toolset's 1.4 is another's 40. A hundred-point misunderstanding with no notation to warn you.
✓Ruled once in the metric store, rendered with its unit, computed one way everywhere.
The calendar fork. 4-5-4, 4-4-5, or thirteen periods, silent in the data, shifting every comparison that crosses it.
✓The calendar is a recorded ruling, and period labels carry it.
Two honest headlines from one file. "Latest complete period" at quarter grain or month grain yields figures five times apart from the same rows; both builders were being reasonable.
✓Granularity is pinned to what the file itself claims, and an unattended choice is a visible PROPOSED ruling.
The definition that drifted between copies. The same reference table pasted into three scripts, updated in two. The stale copy keeps reporting, correctly, from the wrong facts.
✓The metric store's no-drift law: one stored formula, either executed directly or re-derived by a check that fails naming the key.
The average of incomparables. Two retailers define the same metric differently; a portfolio view averages them into a number that describes nothing.
✓Principle 10: definitions never travel across sources; the portfolio view shows the fork instead of averaging it away.
The verification is theater
The class we hunt hardest, because it does not just miss errors, it manufactures confidence. Most entries here came out of our own audits, which is to say: we built them, caught them, and wrote the law.
The verify script that silently did nothing. The agent wrote itself a verification script; it no-op'd; everything passed. The wrong numbers it blessed were headed for an executive deck.
✓Principle 6: every guard needs a fire test that feeds it known-bad input and watches it block.
The check that cannot fail. We caught a shipped check that was literally pass-if-true, rendered green on four surfaces beside real gates, indistinguishable to any reader.
✓A rendered check row without a fire test that can turn that exact row red may not render as PASS. The shipped doctrine gate enforces it mechanically.
The painted row. Verdict text hardcoded to green while the scan's actual findings were computed and discarded.
✓Same law: a row that cannot change is deleted, not rendered. INFO exists for rows that inform without guaranteeing.
Fire-test theater. The test re-implements the guard's comparison inline and stays green forever while the deployed guard is broken.
✓The fire test must call the deployed guard through the production code path. Evidence, not assertion.
Borrowed evidence. Two checks pointing at the same fire-test log, proving neither.
✓The gate requires one evidence file per check, naming that exact check, showing the captured red run.
Both rails from one blueprint. A "two independent reads" verification that built both reads from the same field list and compared 537 values to themselves. Agreement guaranteed, meaning zero.
✓Independence goes all the way down: shared parse logic must be disclosed and counted as single-read.
The tolerance that ate the error. Two reads disagreed, so the tolerance grew until they did not.
✓Principle 2: you investigate disagreement; you never widen the tolerance to make it go away.
The screen calibrated by its own target. An outlier ceiling derived from the data being screened, so the 100x slip became its own ceiling and certified clean.
✓Principle 11: the tier basis comes from outside the screened data, or the screen reports itself UNCALIBRATED, visibly.
Detection without disclosure. The check table counted the flag. The certified page beside it showed the figure clean.
✓A flag travels the full disclosure path or the build fails: stakes, what-if, and the glyph at every render, aggregates included.
The tie that compared revenue to seats. A green "ties" mark asserting agreement between two numbers that measure different things.
✓Every tie names its two sides and their units; a tie between incomparables is a failed check, not a green one.
The gate that certified the leak. A round-trip check blessing a board PDF that carried the build machine's temp path on every page.
✓Deliverable hygiene is a gated law: timestamps and local paths fail the build before anything ships.
The hash that disproves itself. A manifest sha256 computed in text mode, failing the customer's first integrity check on the very artifact meant to prove integrity.
✓Hashes are computed from raw bytes and re-verified by an independent read before the build may pass.
The stale fingerprint. An index page displaying truncated hashes from an earlier cut of the files beside it.
✓The gate checks every hash claim on every surface, prefixes included, against the shipped bytes.
The fix that landed on the letter. A rule names a mechanism; the mechanism gets repaired; the harm the rule exists to prevent goes on landing. Our own round-four fixes did exactly this, and round five caught it.
✓Every law is now tested against the harm, not against its own wording. A fix that cannot be shown stopping the original failure is not a fix.
The check that reads its own configuration. A check with a genuine fire test, a unique evidence file, and a clean gate run, whose verdict is nonetheless built from a config flag the same build set to true. It cannot see the render step drop a marker.
✓The check reads the shipped artifact, never the configuration that produced it. A check that consults its own settings is a mirror, not a gate.
The check that proves a coincidence. A guard built from literals that hold for one file: one region, one invoice id, one month. It reddens honestly, and it proves nothing about the next export.
✓A check is written against the property, not the dataset. If moving a row between groups makes a correct build fail, the check encodes trivia.
The honest refusal that catches nothing. Told not to calibrate a screen from the data it screens, three builds refused to calibrate at all. Every one reported itself uncalibrated, honestly, and a hundredfold quantity slip certified as cleanly as before.
✓The screen has an always-available non-circular basis: compare each value against its group with that value excluded. A slip cannot move a distribution it is not part of, and an unscreened column is never called certified.
The wrapper betrays the numbers
The figures can be perfect and the deliverable still dies on first forward. A pack a buyer cannot send onward is not finished work.
The certified page that opens as garbage. No charset declaration, so every heading renders as mojibake in the recipient's browser.
✓Doctype and charset in the first bytes of every artifact, gated, with a render check for replacement glyphs.
The board pack that names your temp folder. Browser print chrome stamping the build timestamp and a local file path across all 25 pages of the PDF.
✓Print chrome is suppressed; header, as-of date, and page numbers come from the page's own print CSS, and leaks fail the gate.
The manifest that contradicts its page. Hand-typed counts in the summary disagreeing with the machine output beside them: says 3, shows 6.
✓Principle 1 extends to every shipped artifact; a prose figure absent from machine output fails the gate by grep.
The stack trace deliverable. One unparseable date, a raw exception, zero artifacts, and 537 rows of provable verdicts discarded with it.
✓Principle 5: a refusal is a report. The BLOCKED page ships every verdict earned before the stop, and a bare abort is itself a failed check.
The rerun that trampled the good pack. A failed build writing over the last known-good deliverable file by file, leaving a certified-looking ruin.
✓Builds assemble in staging and swap only after the final gate passes; the prior pack survives byte-for-byte.
The refresh time bomb. The reporting window's end date hardcoded in five places; next month's run is silently last month's.
✓Principle 8: dates, bounds, and windows are configuration in one place, and an empty setting fails loudly.
The step nobody can run again. The first transformation, raw file to working data, done interactively and never preserved, so no number is reproducible from source.
✓Provenance and reproducibility: every figure traces to a script that runs from the archived raw bytes.
The estimate outside its own interval. Two code paths applying different exclusions, until every headline KPI sat outside its own published confidence interval.
✓One source of computation per figure, and a fire-tested check that re-derives what the page displays.
The blocked run that ate the good pack. A build hit its blocking condition, cleared the output directory, and moved a failed run into its place, destroying the last certified pack. The page it left behind said the previous pack was untouched.
✓Builds assemble in staging and swap only after the final gate passes, so a failed run leaves the prior pack byte-for-byte intact. The claim and the filesystem agree or the build fails.
Why we can write this list
Kymira did not start as a product. It started as a real reporting system built from large, messy exports, where roughly 90% of a fourteen-hour build was verification, and six plausible, wrong numbers were caught on their way to a leadership deck. The rules that caught them became the doctrine. Then we turned the method on ourselves: four rounds of adversarial audits, agents attacking booby-trapped data with the skills alone, graders hunting what survived. Most of the theater entries above are our own failures, found in our own builds, and closed with laws and a mechanical gate before any customer met them.
That is the trust argument in one line: we know what a working tool is because we have catalogued, in public, exactly what is not one.