GUIDE · VERIFICATION
How to verify a number an AI produced: a seven-check protocol.
An AI can produce a number that is confidently, invisibly wrong: the arithmetic ties, the page looks reviewed, and nothing on it is false-looking. Verification is a fixed sequence of checks, not a feeling about the output. Seven checks, in order, each one doable by hand on your own export.
Why a number that looks reviewed is not a number that was checked
Polish is the signal readers have always used for care, and it is the one signal AI output supplies for free. Robert Sahlin named this the aesthetic credibility trap in a post dated 28 August 2026: "The polish is indistinguishable from correctness." Garbled output announces its own failure; a clean report does not. That asymmetry is the entire reason a protocol has to exist: you cannot detect this class of error by looking harder at the result.
Scale, from a third party rather than from us: Databox's survey of more than 100 business users, run in June 2026, reports that 74 percent had shipped or shared something built on a generative AI number that turned out to be wrong (source, retrieved 2 September 2026). We have not audited that survey's method; it is cited as a third party's published finding, not as our own measurement.
The seven checks, in order
Run them in this order because each one narrows what the next has to consider. A figure that passes all seven is verified in a narrow, stateable sense. A figure that has passed none is a plausible sentence.
- Tie the number to a total the file itself states.
- Read the same number a second way that shares no logic.
- Prove the check can fail before you believe it passed.
- Take the period from the contents, never the filename.
- Put the definition beside the number, not in a footnote.
- Keep provenance back to the source bytes.
- Refuse rather than guess when the file is ambiguous.
01Tie the number to a total the file itself states.
The only check a vendor cannot argue with is the file's own printed total. Sum your rows and compare against the totals row, the stated grand total, the printed base, whatever the export asserts about itself, not against a second sum you computed the same way. A number that ties to nothing is unverified, and should be labelled unverified rather than presented plain. Counts get no tolerance: a count matches or it does not, and a default tolerance of 1.0 absorbs exactly the dropped row the check exists to catch.
Open the export and find the row it
prints as Total, or the grand total in the block above the header. Sum the data rows
separately and subtract. If the export prints no total anywhere, write no anchor
beside the figure and treat it as unproven rather than quietly accepted.
skills/anchor-discovery interrogates an unfamiliar export and proposes the anchor before any number is computed, and it refuses a circular one: a total derived from the rows it checks can never fail. skills/verification-practice turns that proposal into a check that reruns on every build. Doctrine principle 1.
In our own audits: a section
carrying a Total beside the three rows it already sums, so the block sums to
139,076.10 where the file states 69,194.63. Both computation rails agreed perfectly on the
wrong number. Only the file's own printed total saw it.
02Read the same number a second way that shares no logic.
One careful read is one point of failure. Compute the value twice by two routes that share as little as possible: once by where the value sits in the file, once by the label printed beside it. Different location logic, different parsing route. When the two disagree, that is a refusal, not a warning, and you never widen the tolerance until they agree. Know the limit and state it: two readers of the same cell prove placement, not value, and two readers located by the same labels prove arithmetic, not the section.
Ask for the figure a second time in a fresh session, described differently: by column position once, by header name the other time. Then compute it yourself in the spreadsheet. Three routes and one answer, or you have found something worth reading before anyone acts on the number.
skills/verification-practice builds two rails and prints which degree of independence each figure actually earned. Its hard rule: independence goes all the way down, so if both paths call the same parse function for a field, the page says so and counts that field as single-read. Doctrine principle 2.
In our own audits: a unit check that ran one direction only. The percent suffixes were stripped, both paths agreed, and the engine published a figure that was off by a factor no reader would have caught, because both rails shared the same blueprint. Agreement between two rails that share a flaw is not evidence, it is a louder version of one read.
03Prove the check can fail before you believe it passed.
A green check is a claim about a mechanism you have not tested, and the dangerous check is the one that can never trip. So break something on purpose: take a copy of the file, change one figure so the total no longer ties, and run the same verification again. If it still passes, the check was decoration and every green mark beside it was worthless. The sabotage has to reproduce the actual defect through the same code path the real run uses, not a neighbouring case and not a re-implementation of the comparison.
Duplicate the export. Add 1,000 to one cell in a summed column. Hand the copy back with the identical request. A verification that reports the same clean result on both files has verified nothing, and you have learned that in two minutes instead of at a board meeting.
skills/verification-practice requires a sabotage case per guard; skills/adversarial-review runs the finders and skeptics loop that catches what reading code cannot. The shipped gate/doctrine_gate.py enforces it mechanically: a row with no fire-test evidence may not render as PASS. Doctrine principle 6.
In our own adversarial audits,
in a single round: a check reading "pass" if True else "fail" rendered green on
four surfaces, and a two-rails verification that built both rails from the same field list
compared 537 values to themselves. Both sat beside real gates, indistinguishable to a
reader.
04Take the period from the contents, never the filename.
A file called weekly_sales.csv may hold last week, another account, or both.
Vendor exports are named by the system that produced them. Find the period inside the bytes:
the pre-header block, the embedded title, the report metadata, or the minimum and maximum of a
date column. Then separate three dates that look like one: the coverage window the figures are
for, the as-of instant a balance was true, and the run stamp recording when someone pressed
export. Only the first two can prove a period.
Read the first ten lines above the header row before anything else. Then sort the date column and read its first and last value. If the only date in the file sits beside the word Generated, Printed, or Run date, you have the export date and no period at all: say so on the page.
skills/period-freshness gives every date a role (coverage_start, coverage_end, as_of, run_date), drops a labelled run stamp before measuring period, and states that it did. skills/hostile-exports mines the pre-header block the rest of the toolchain discards. Doctrine principle 4.
In our own audits: a period gate that took the minimum and maximum across a year-over-year comparison window, so July figures published under a November heading and the gate recorded PASS. The November date really was in the file. The run stamp is also the one date a stale re-export always has fresh.
05Put the definition beside the number, not in a footnote.
Most silent wrong numbers are wrong with a right value: the arithmetic ties, every gate
passes, and the word on the label means something other than what the reader assumes. Churn,
bookings, revenue, sell-through and active subscriber all carry several live definitions inside
one company, and every one of them reconciles perfectly against its own source. So decide which
definition this is before computing, then print it beside the value:
Bookings $2.4M (order-entry date, gross of cancellations). A definition in a
tooltip nobody opens is not disclosure.
Write one sentence saying what the number counts, over what window, and out of what base. Ask a colleague in another function to write the same sentence without seeing yours. If the two differ, the figure is not ready to leave the room, whatever the arithmetic says.
skills/contested-metrics
finds the fork, asks in business language with both numbers attached, and records the answer
as a ruling. skills/metric-store keeps that ruling in a versioned
metrics.yaml the agent consults before computing anything named. The
Metric Library holds 351 metric definitions, with the fork named on
each one. Doctrine principle 10.
At Meridian Journal Group, our fictional tenant: a subscription footer stated 712 cancellations while the file held 936 stops. The 224-row gap was involuntary churn, failed cards and expiries, that the column silently excluded. The column name told you none of that.
06Keep provenance back to the source bytes.
A number is reproducible when a stranger who trusts nobody involved can re-run it and get the same answer. That needs a record: the content hash of the input file, the version of the code that built it, the period and parameters asked, the values produced, and the verdict each check returned. Hash the file the moment you receive it, and never let a later download overwrite those bytes under the same name. A hash pointing at bytes that have since changed verifies against nothing and is decoration.
Run shasum -a 256 on the
export and paste the result into the report beside the date. Keep the original under a
filename you never reuse. When the figure is questioned six weeks later, you can prove which
bytes produced it instead of arguing about which download it was.
skills/provenance-reproducibility writes the record and the one command that runs it back, checking identity first so a hash mismatch stops the report rather than blaming the engine. skills/trust-the-upload makes the stored bytes immutable, with collisions checked against the filesystem and every name in the history.
In our own audits: stored input bytes were overwritable by a same-name re-upload, so a provenance record pointed at bytes that no longer existed. The hash still verified, against nothing anybody could reproduce. Content-address the store and the overwrite becomes structurally impossible.
07Refuse rather than guess when the file is ambiguous.
Guessing past ambiguity is how wrong numbers get published, and the guess is invisible in the output. A label that appears zero times or twice is an error. A missing column is an error. A date that parses two valid ways is as much an error as one that parses none: 4/9/2026 is April 9th or September 4th depending on a locale the file never states. The correct response is to stop and say precisely what was ambiguous. Refusing is not the run failing, it is the run working.
Say in writing that ambiguity is a question for you and never a default, then look for the tell: a figure that arrived without a stated assumption where the file was genuinely unclear. Ask what happened to the rows that did not parse. Silence there is the answer.
skills/anchor-discovery runs on one directive: you propose, the human confirms, the engine enforces. skills/verification-practice gives every refusal a machine-readable exit code, so a pipeline reads codes rather than prose. Doctrine principles 3 and 5: stale and correct beats fresh and wrong.
In our own audits: one ambiguous date in an appended row crashed a build with a raw error and zero artifacts, on a file whose other 537 rows had already earned fourteen passing verdicts. A refusal has to be an artifact, a page that says what is blocked and what the file still proves, not a stack trace.
Which skill mechanizes which check
Every check above is a thing a person can do by hand once. The skills exist so it happens on every rebuild, in the same order, without anybody remembering to ask.
| Check | Doctrine principle | Skill folder |
|---|---|---|
| 1. Tie to the stated total | 1: an AI never authors a number | skills/anchor-discovery, skills/verification-practice |
| 2. Second independent read | 2: two independent reads beat one careful one | skills/verification-practice |
| 3. Fire test the check | 6: test that a check fires, not that it exists | skills/verification-practice, skills/adversarial-review, gate/doctrine_gate.py |
| 4. Period from contents | 4: filenames lie | skills/period-freshness, skills/hostile-exports |
| 5. Definition beside the number | 10: a metric without its definition is not a number | skills/contested-metrics, skills/metric-store |
| 6. Provenance to bytes | What a certified deliverable is | skills/provenance-reproducibility, skills/trust-the-upload |
| 7. Refuse, do not guess | 3: refuse to guess · 5: fail closed and stale | skills/anchor-discovery, skills/verification-practice |
All eleven principles are published in full at kymira.ai/doctrine. The failures each one closes are catalogued at kymira.ai/failure-modes, and the export shapes that produce them are in the Format Library.
Not for: what the seven checks do not prove
Stating the limit is part of the protocol. A check whose boundary is unstated gets read as a guarantee it never gave.
Questions
How do I know if the numbers Claude gave me from my CSV are right?
You do not know from reading them, which is the whole problem. Run the seven checks in order: tie the figure to a total the file itself states, read it a second way that shares no logic, prove the check can fail by feeding it a broken copy, take the period from the file's contents, print the definition beside the number, keep a hash back to the source bytes, and refuse rather than guess where the file is ambiguous. A number that has passed all seven is verified in a narrow, stateable sense. One that has passed none is a plausible sentence.
Can I just tell the AI to double-check its numbers?
You can write the sentence, and it helps. What a sentence cannot do is cover the cases it does not name: a Total row sitting in the same columns as the rows it sums, a run stamp standing in for a coverage window, a check that structurally cannot fail, a second read that shares a parse function with the first. Those are the several hundred rules underneath the sentence, and each one is there because a real number went out wrong.
Which of the seven checks is fastest to do by hand?
The first one. Open the export, find the total the file prints about itself, sum the data rows separately, and subtract. That is about two minutes on most files, and it catches the commonest failure there is: a Total or subtotal row sitting in the same columns as the rows it sums, quietly counted a second time by anything that reads top to bottom.
Start with the free skill
The seven checks are the method. The skill named honest-dataviz is free forever and published in full, so you can read exactly how a rule is written before deciding whether the rest is worth anything to you. It ships with all eleven doctrine principles and the install guide, and it needs no email. Nothing you run it on leaves your machine, which is the whole of the security model. If you want the seven checks running on your own exports, the quick start has the prompts.