Requirement
The buyer specified reward-bearing environments for coding-agent RL, not a static benchmark. That distinction set the acceptance criteria.
| # | Requirement | Acceptance condition |
|---|---|---|
| R1 | Not solvable by recall | Task remains unsolved by a model reproducing the closest public library |
| R2 | Deterministic reward | Same submission, same score, every run |
| R3 | Dense signal | Partial progress scores proportionally rather than collapsing to zero |
| R4 | Offline grading | No network path from the graded environment to a package index |
| R5 | Auditable | Buyer can re-run every claim we make about a task |
| R6 | Sustained volume | Hundreds of accepted tasks per week, not a one-time drop |
R1 eliminated the existing options.
Why existing task sets failed R1
If a task asks an agent to implement a library that sits on a package index, whose source is on GitHub and whose documentation is in every crawl, a strong model does not reason from the specification. It reproduces what it already holds.
The task still returns a score. The score measures retrieval.
As an evaluation that inflates the number. As an RL environment it is worse, because the reward attaches to retrieval and the policy optimizes accordingly. The usual mitigations do not hold: recency fails because crawl date is not training cutoff, obscurity fails because obscure code is still crawled, and holding out the test suite fails because the model is not recalling the tests, it is recalling the implementation the tests describe.
Task construction
Every task is a natural-language specification for a library in one of two classes.
| Class | Count in reference tranche | Construction | Gate available |
|---|---|---|---|
| Invented domain | 4 | No public analogue exists: a version scheme that is not SemVer, unit coercion with banker's rounding and snap-to-zero, an inverted-precedence query DSL, a base-36 check-character codec that is not Luhn | None needed |
| Mutated domain | 8 | Familiar domain, public surface renamed, semantics deliberately inverted | Real upstream library |
Mutation is behavioral, not cosmetic. Renaming functions alone is defeated by a model that reads the specification and maps the new names onto remembered behavior. The rules underneath differ, so recall produces a plausible implementation that fails specifically on the behaviors the specification defines.
The inversion pattern, shown without naming tasks: a Roman-numeral analogue that bans subtractive notation; an ordered-set analogue whose add returns the element rather than an index and where indexing raises; a query language whose disjunction binds tighter than its conjunction.
The solver receives exactly one file per task. The specification. Nothing else.
Grading contract
Each task carries a machine-readable contract rather than a convention.
| Field | Type | Purpose |
|---|---|---|
import_name |
string | Module the structural check imports before any behavioral test runs |
public_symbols |
string[] | Symbols that must exist; a memorized upstream module fails here first |
grader.min_passed |
int | Exact held-out assertion count, counted from the regenerated suite |
grader.pytest_paths |
string[] | Held-out suite roots, resident outside the solver-visible tree |
reference_install |
path | Authored correct implementation |
known_bad_install |
path | Plausible-but-wrong package |
upstream_sdist |
string, nullable | Real upstream tarball, plagiarist gate only; null for invented domains |
min_passed is counted, not estimated. It ranges from 30 to 193 per task and totals 1,308 across the twelve-task reference tranche.
The suite is regenerated from the authored reference, so it asserts the mutated behavior. No upstream suite is ever vendored in. A vendored suite agrees with a memorized package by construction, which would reintroduce the exact failure the lane exists to remove.
Execution environment
| Control | Implementation | Closes |
|---|---|---|
| Network isolation | Container runs with the network namespace off | Fetching the real library at grade time |
| Index isolation | Installs run with a no-index flag | Resolution through a configured mirror |
| Requirement strings | Bare package names refused rather than resolved | The hole where an install by name silently fetches upstream |
| Install source | Path-only | Grader scores only source the candidate wrote |
| Copy guard | Similarity check against reference and upstream, configurable hard-fail | A leaked or copied answer |
Verification gates
Four gates run per task before it ships. A task failing any gate does not go out.
| Gate | Method | Pass condition | Reference tranche |
|---|---|---|---|
| G1 Reference | Authored implementation run through the grader | Passes at the full ship bar | 12 of 12 |
| G2 Known-bad | Deliberately wrong package installed and graded | Installs, clears structural check, then fails behaviorally | 12 of 12 |
| G3 Plagiarist | Genuine upstream distribution run through the task's own grader | Fails | 8 of 8 gated tasks |
| G4 Isolation | Verifier asserts solver-visible tree contents | Contains specifications and nothing else | 12 of 12 |
G2 is the gate most commonly skipped. Without it, a grader may be checking that correctly named functions exist rather than that they behave correctly, and that defect stays invisible until something deliberately wrong is scored against it.
G3 is the claim made executable. Contamination-safety is normally argued in a paragraph. Here the real library is installed into the task environment, run through that task's own grader, and required to fail. The buyer re-runs it.
Scoring
Score is the fraction of held-out assertions passed, so an honest eighty-percent implementation scores 0.80 rather than zero, which satisfies R3. A task counts as passed only at the full ship bar, which keeps the same artifact usable as a binary evaluation without rebuilding it.
Delivery
| Property | Value |
|---|---|
| Accepted rate | 300-600 tasks per week against a 500/week target |
| Acceptance | Measured by the buyer, not self-reported |
| Monthly volume on hand | 1,200-2,400 tasks |
| Format | Docker image, JSONL task index, pytest suite, patch |
| Licensing | Held-out graders licensed separately from solver-visible specifications, so an evaluation split can be withheld from a training split |
What is withheld
Task identities, specification text, and mutation rules are not published. Publishing them contaminates environments the buyer paid for, which would invert the premise the lane is sold on.
Structure is published, as above. Answers are not.
Stated limitation
A grader proves a task is scorable. It does not prove a task is hard.
Difficulty here is structural rather than tuned: because every library is invented or mutated, a model has to implement the specified behavior with no pretraining crutch available. G3 ships the evidence for that claim. It is a different and more checkable thing than a difficulty number we selected ourselves, and it is not presented as more than that.