Failure catalogue · The file lies · 12 of 51
OBSERVED FAILURE MODE
The encoding that crashes the pipeline.
A non UTF-8 export meets a reader that assumes UTF-8. The lucky outcome is a crash; the dangerous one is a half parse that returns rows with a mangled first cell, a code turned into scientific notation, or leading zeros gone, and then carries on to compute totals over whatever survived the decoding.
What we saw
A distributor export at Meridian Journal Group arrived carrying a byte order mark its three sibling files did not have, and one file in the set killed the reader mid-run. That failure was the good case. The instructive one came from the same family: a file that decoded far enough to produce rows, with a mangled first cell and account codes stripped of their leading zeros, so the join against the reference table matched fewer accounts and the totals it produced were sums over the survivors. Nothing in the output said a byte had been lost.
Why it passes a glance
A half parse produces rows, and rows look like success. The totals computed from them tie against each other, and a reader who does not know how many accounts should have matched cannot tell that some are missing. Mojibake is visible only where a human reads a name, and most of a report is numbers. The crash, by contrast, is loud and gets fixed.
What addresses it
The trust-the-upload skill fixes the order of the landing pass: archive the raw bytes untouched, then sniff the container and prove the encoding before any value is read, so nothing downstream ever guesses at a decoding. Doctrine principle 3, refuse to guess, governs what happens next: a file that will not decode cleanly is an error to name and stop on, not a list of encodings to try until one returns rows.
Check your own file in two minutes
- Look at the first bytes of the file for a byte order mark, and check the line endings, before opening it in anything.
- Read a sample of rows containing names or addresses and look for replacement characters and mojibake.
- Compare the row count and the count of distinct keys against the sibling exports from the same system.
- Check any code column for lost leading zeros and for values rendered in scientific notation.
What this does not catch
Proving the encoding proves the bytes decoded. It says nothing about values a spreadsheet already damaged before the file reached you, such as a code column stored as a number by whoever opened the export and saved it again.
Quick answers
- Is a crash the worst case?
- No, a crash is the case you find out about, while a half parse publishes totals computed over whatever survived the decoding.
- Why archive the raw bytes first?
- Because every later question about the file, including which encoding it actually used, is answerable only from the bytes as they arrived.
- Do sibling exports share an encoding?
- Not reliably: one file in a set can carry a byte order mark or different line endings that its siblings do not.
Nearby failures
Last updated 2026-09-02 · Dylan, founder · one of 51 observed failure modes, every one seen in a real build or in our own audits, none invented.