A de-identification pipeline that reports zero findings is reporting one of two things. Either the corpus is clean, or the scanner is checking the same assumptions the pipeline was built on. These are very different situations and they produce identical output.
We have watched a corpus pass every automated check we had and then fail badly under independent human review. Not marginally. The review returned defects in the dozens, high severity, on material that was already delivered. Nothing in the pipeline was broken. It was answering the wrong question.
The checker and the scrubber share a mental model
An automated checker is written by the same people who wrote the scrubber, usually in the same week, with the same mental model of what identifying information looks like. It therefore searches for the categories the scrubber already handles.
Names, emails, phone numbers, account identifiers. The scrubber removes them, and the checker confirms they're gone. Both are correct. Both are looking at the text layer of a document, which is where the scrubber operates, and neither is looking at the document as a thing a human will open.
The result is a verification step that validates the build's own beliefs rather than the delivered artifact. It isn't a weak check. It's a check of the wrong object.
The six classes that get through
The leak classes that get through share a property: they're identifying without being text the scrubber was pointed at.
Hyperlink targets. The visible text gets renamed. The href underneath doesn't. A scheduling link, a profile URL, a document share link, each carrying a real name in the path, sitting invisibly under text that reads as anonymous.
Images of people. A team page with the names replaced and the faces intact is not de-identified. Neither is a signed document with a scanned signature, or a screenshot with a visible account bar.
Brand marks and domains in the page layer. Text-layer renaming leaves logos, letterhead, footers, and watermarked domains untouched. The document now says one company name and looks like another.
Recipient and licensing stamps. Research and legal documents often carry a per-recipient stamp repeated on every page. It identifies the original licensee, it isn't part of the body text, and it survives a body-text scrub across a hundred pages at a time.
Regulated data inside ordinary documents. Health information in an insurance workflow, financial identifiers in an onboarding packet. The document genre reads as routine and the content is not.
Third-party identities. Scrubbers are built around the principal. Counterparties, signatories, and their addresses are identifying too, and they're usually somebody else's data rather than yours to release.
Scrubbing harder makes it worthless
The obvious response is to scrub harder, and it's wrong. A corpus scrubbed until nothing identifying could possibly remain is a corpus with no operational substance left, and operational substance is the entire reason anyone wanted it.
The rule we work to is: remove who it is, never what happens. Identities become stable tokens. Amounts, dates, sequence, structure, and outcomes stay exactly as they were, because those are the signal. A ledger with the numbers blurred is not a safer dataset, it's a worthless one.
That rule only holds if the removal is precise, which is what makes the verification question load-bearing rather than procedural.
Verify the artifact, not the log
The fix is not a longer rule list. Rules are how you got here.
Verification has to read the shipped bytes rather than the build's record of what it did. Open the delivered file the way a recipient will. Extract the links and look at the targets. Look at the images. Render the pages and read what is actually on them, including the parts that aren't body text. Check the artifact after packaging, not the intermediate state before it.
And then have a human read a real sample of it. Not a spot check on the first three files, a genuine read, by someone who didn't build the pipeline and is not invested in it passing. This is the layer that finds the class of leak nobody wrote a rule for, and it's the only layer that reliably does.
Independence matters more than volume here. A reviewer who knows how the scrubber works will unconsciously check the things the scrubber handles.
Human-written is necessary and not sufficient
Two adjacent findings came out of the same process and both became hard rules.
Being human-written is necessary and not sufficient. A templated document produced by a person is still a template, and a corpus of them teaches format rather than work. Quality has to be scored within genre against a rubric, with template subtraction as the controlling test, or you end up measuring length and calling it substance.
And suspected model-generated material should be excluded rather than scored down. In a corpus whose entire value proposition is that it's real professional work, a probabilistic quality penalty on synthetic-looking material is the wrong instrument. The answer is a hard exclusion with the exclusions published, so a buyer can see what was taken out and why.
What it costs
Every corpus we ship now goes through an independent human read before delivery, and it isn't cheap. It's also not optional, because the alternative is discovering the defect after the buyer does.
If a data vendor tells you their de-identification is fully automated and reports zero findings, the correct response is to ask what the checker reads. If the answer is the pipeline's own output, you have learned what the pipeline believes about itself.