Failure catalogue · The metric means something else · 24 of 51
OBSERVED FAILURE MODE
The definition that drifted between copies.
A metric definition pasted into more than one script becomes several definitions the moment one copy is updated. The stale copy keeps reporting correctly from the wrong facts. Its arithmetic is sound, its tie to source passes, and the figure it publishes belongs to a rule the business has already retired.
What we saw
In our own audits, one reference table for a named metric lived in three scripts. A definition change landed in two. The third kept running on schedule, tying to its own source, publishing a figure under the same label as the updated pages. Nothing failed anywhere. The two figures differed by whatever the change was worth, and the only way to see the drift was to read all three scripts side by side, which nobody does once a change is believed to be finished.
Why it passes a glance
A stale copy is not broken code. It computes exactly what it was told to compute, ties to its own source, and reproduces on every run. Every check inside its build is green, because the build has no way to know a decision was taken elsewhere. The contradiction is visible only between artifacts, never inside any one of them.
What addresses it
The metric-store skill gives every metric one stored formula with its anchor, unit, and history, and enforces a no-drift law: either the stored formula is executed directly, or a fire-tested check re-derives each rendered figure from the store and fails naming the key when the two disagree. That is Principle 10, a metric without its definition is not a number, made mechanical.
Check your own file in two minutes
- Search the workspace for every place the metric's formula appears, including notebooks and one-off scripts.
- Compare each copy against the definition you believe is current.
- Delete the copies and point every build at one stored entry.
- Add a check that recomputes a rendered figure from the stored formula and fails naming the metric.
What this does not catch
One stored definition stops copies drifting apart. It does not tell you which copy was right when they already disagree, and it cannot restate archived pages built under an older rule. Old reports stay correct under their own definition, which is why definitions are versioned rather than edited in place.
Quick answers
- Why not just fix all the copies?
- Because the next change has the same problem. The fix that lasts is one stored definition the build reads, so a change lands once and every dependent page rebuilds from it.
- Should old reports be rebuilt when a definition changes?
- Not silently. Last quarter's report was correct under last quarter's definition. Add a superseding entry, keep the old one, and mark the changeover on every affected chart so the discontinuity is explained.
- How would I notice this today?
- Compare the same metric across two pages built at different times. If the figures differ and both tie to their own sources, look for a second copy of the formula before looking for a data problem.
Nearby failures
Last updated 2026-09-02 · Dylan, founder · one of 51 observed failure modes, every one seen in a real build or in our own audits, none invented.