Every coding benchmark published after 2023 comes with a paragraph about contamination. The paragraph explains that the authors were careful, that the tasks are novel or recent or held out, and that leakage is therefore unlikely. Then the benchmark reports a number.
The paragraph is an argument. What it should be is a test.
Retrieval scores the same as reasoning
If a task asks a model to implement something that exists publicly, a sufficiently large model doesn't have to reason about the specification. It reproduces the artifact it has already read. The task still produces a score, and the score is still real in the narrow sense that assertions passed. It just measures retrieval rather than the capability the benchmark is named after.
For evaluation this inflates the number. For reinforcement learning it's worse, because the reward is now attached to the wrong behavior. A policy trained against a recall-solvable environment learns that recall is what pays.
The usual defenses are weaker than they look. Recency fails because the crawl date is not the training cutoff and never was. Obscurity fails because obscure code is still code and still gets crawled. Held-out test suites fail if the implementation being tested is public, since the model is not recalling the tests, it's recalling the thing the tests describe.
The gate
There's a direct test available and almost nobody runs it.
Keep the real upstream library. Install it into your own task environment. Run it through your own grader. Require it to fail.
That's the whole mechanism. If the genuine public implementation of the thing your task asks for passes your task, then your task is passable by recall and you have proved it against yourself. If it fails, you have shipped evidence rather than an assurance, and any buyer can re-run it.
We keep the real upstream distributions in the private grading tree for exactly this reason, and the gate runs before anything ships. It isn't a claim in a README. It's a check in the verifier, and a task that doesn't clear it doesn't go out.
You need a domain that recall can't reach
A gate you can't pass is not useful, so the task construction has to give you something to pass it with. Two constructions work.
The first is to invent the domain outright. A version scheme that isn't SemVer, a unit-coercion library with its own rounding rules, a query language with deliberately inverted operator precedence. There's no upstream to gate against because there's no upstream, and the model has to read the specification because there's nothing else to read.
The second is more interesting: take a familiar domain and mutate it. Rename the public surface so a memorized module doesn't even import under the name the structural check requires, then invert the semantics so a solver that remembers how the domain works is wrong on the graded long tail. A Roman-numeral library that bans subtractive notation. A set type whose add returns the element rather than an index and where indexing is a type error. An inflection library whose plural rules aren't English.
The mutation has to be behavioral rather than cosmetic, and this is where the approach usually gets done badly. Renaming functions alone is defeated by a model that reads the specification and maps the new names onto remembered behavior. The rules underneath have to actually differ, so that recall produces a plausible implementation that fails specifically on the behaviors the specification defines.
The gate doesn't prove your grader works
The gate proves the task is not recall-solvable. It doesn't prove the grader works. Two companions cover that.
A known-bad implementation should install cleanly, pass the structural symbol check, and then fail the behavioral suite. If it passes, your grader is checking that functions exist rather than that they're correct, which is a surprisingly common defect and completely invisible until you test for it.
And the suite has to be regenerated from your own reference rather than vendored from upstream. A vendored suite asserts upstream behavior, which means it agrees with a memorized package by construction. You'd have carefully mutated the domain and then shipped a grader that rewards the unmutated answer.
None of this survives network access
None of this survives a grader with network access.
If the environment can reach a package index, a model can install the real library and have it scored, and every gate above becomes decorative. Grading has to run with the network namespace off, installs have to be path-only, and bare requirement strings have to be refused rather than resolved. That last one is the specific hole worth naming: a grader that accepts a package name as an install target will happily fetch and score the upstream implementation, and it will look like a passing run.
The question to ask
If someone sells you evaluation tasks or RL environments, the question is not whether they considered contamination. It's whether they can show you the real implementation failing their grader.
If they can, the claim is checkable and you should check it. If they can't, you're buying the paragraph.