CODE RL ENVIRONMENTS
Repo Build & Repair Tasks
Natural-language specifications and real issues turned into runnable repositories with held-out test suites, each verified to fail before the fix and pass after it.
Inspect a real sample
Representative records in the delivery format, ready to inspect before licensing the full dataset.
Grading contract for one environment
Real field shape from a shipped task. Identity, symbol names, and mutation rules are withheld here on purpose - publishing them would contaminate a graded environment we sell.
{
"task_id": "05-<withheld>",
"kind": "mutated-domain",
"import_name": "<withheld>",
"public_symbols": ["<5 renamed symbols>"],
"grader": {
"pytest_paths": ["."],
"min_passed": 193,
"source": "Regenerated from the authored reference. Asserts the MUTATED semantics."
},
"reference_install": "reference",
"known_bad_install": "known-bad",
"upstream_sdist": "<real upstream tarball, plagiarist gate only>",
"anti_memorization": "Public surface renamed; every rule inverted from the well-known
analogue. A memorized reproduction fails the structural check first and the
behavioral long tail second - verified by the upstream-fails gate.",
"provenance": "Authored from scratch. Upstream is retained only to prove it fails."
}Held-out assertion distribution across the reference tranche
Real counts, task identities withheld. Sums to the 1,308 figure quoted above.
task,origin,held_out_assertions,plagiarist_gate 01,invented,178,none 02,mutated,121,upstream pinned 03,mutated,74,upstream pinned 04,mutated,63,upstream pinned 05,mutated,193,upstream pinned 06,mutated,149,upstream pinned 07,mutated,51,upstream pinned 08,mutated,61,upstream pinned 09,mutated,30,upstream pinned 10,invented,161,none 11,invented,116,none 12,invented,111,none ,,1308,8 of 12 gated
Record shape
Every field, its type, whether it can be null, and a representative value.
| Field | Type | Constraint | Description |
|---|---|---|---|
| task_id | string | required | Stable task identifier within the tranche. e.g. 07-<withheld> |
| kind | enum | required | invented-domain | mutated-domain | mined-issue. e.g. mutated-domain |
| import_name | string | required | Module the structural check imports before any behavioral test runs. |
| public_symbols | string[] | required | Symbols the candidate must expose; a memorized upstream package fails here first. |
| grader.min_passed | int · assertions | required | Exact held-out assertion count, counted from the regenerated suite. e.g. 193 |
| grader.pytest_paths | string[] | required | Held-out suite roots. Never shipped inside the solver-visible tree. |
| reference_install | path | required | Authored correct implementation. Must install and pass at the ship bar. |
| known_bad_install | path | required | Plausible-but-wrong package. Must install cleanly, then fail the suite. |
| upstream_sdist | string | nullable | Real upstream tarball for the plagiarist gate; null for invented domains. |
| anti_memorization | string | required | Written record of every renamed symbol and mutated rule, kept private to the grading tree. |
| provenance | string | required | Authorship and licensing of the reference implementation. |
Spec-to-Repo Tasks
A natural-language library specification plus a held-out assertion suite. The agent writes the library; the suite decides. Every task is validated against a plagiarist gate that proves the answer is not reachable by copying an existing package.
Long-Horizon Repository Tasks
Multi-file, cross-subsystem changes mined from real permissively-licensed repositories - real issue, real fix, real held-out tests - served in a history-stripped environment so the answer cannot be read out of the Git log.
Teacher Trajectories
Full tool-call traces from frontier coding agents solving these environments, with reasoning preserved, graded on the same held-out suites and passed through a Git-history cheat scan.
How it is built
- 01
Author or mutate the domain
Each task is either an invented domain with no public analogue, or a familiar domain whose public surface is renamed and whose semantics are deliberately inverted. A model that reproduces the library it remembers is wrong on the graded long tail rather than merely unlucky.
- 02
Regenerate the held-out suite from the reference
Tests are generated from the authored reference implementation and assert the mutated behavior. No upstream test suite is ever vendored in, so the suite cannot agree with a memorized package by construction.
- 03
Structurally isolate the held-out material
The solver receives exactly one file per task: the specification. Reference, regenerated tests, known-bad package, grading contract, and upstream tarballs live in a sibling private tree, and the verifier asserts that nothing but the specification exists in the solver-visible directory.
- 04
Grade offline and path-only
Grading runs in a pinned image with the network namespace off. Installs use a no-index flag and bare requirement strings are refused outright, so the grader can only ever score source the candidate wrote.
- 05
Prove the grader discriminates
Every task is verified four ways before it ships: the reference passes at the ship bar, the known-bad package installs and then fails, the real upstream library run through the task grader fails, and the solver-visible tree contains nothing but specifications.
- 06
Mine and validate long-horizon tasks
Repository tasks are mined by locating paired source and test commits in permissively-licensed projects, then validating each as a genuine fail-then-pass transition inside a history-stripped environment. Candidates that do not reproduce that transition are discarded.
How we validate
What each evaluation measures and how it is run. Where no benchmark is published, we show the methodology and say so.
Plagiarist gate
Measures
Whether reproducing the real public library passes the task.
Method
The genuine upstream distribution is installed into the task environment and run through the task own grader. The gate requires it to fail.
Result
Real measured: upstream fails on every mutated-domain task in the reference tranche. The four invented-domain tasks have no public analogue to gate against.
Known-bad discrimination
Measures
Whether the grader scores behavior rather than mere presence of the right symbols.
Method
A plausible-but-wrong package is required to pass the structural symbol check and then fail the behavioral suite.
Result
Real measured: passes on every task in the reference tranche.
Held-out assertion count
Measures
How much graded surface each task actually carries.
Method
Assertions are counted directly from the regenerated suite and recorded as the exact pass threshold in the grading contract.
Result
Real measured: 1,308 held-out assertions across the 12-task reference tranche, ranging from 30 to 193 per task.
Copy guard
Measures
Whether a submission is byte-near the shipped reference or the real upstream source.
Method
A similarity check runs on every submission with a configurable hard-fail threshold, as defense in depth behind the network isolation.
Result
Methodology-stage. Reported per submission; no aggregate false-positive rate is published.
Ground truth
What correct means for this data, and how it is established.
Ground truth
The authored reference implementation and its regenerated held-out suite. For mined repository tasks, the real upstream fix commit and the tests that commit made pass are the ground truth.
How it is established
A structural symbol check runs first, then the held-out suite. Score is the fraction of held-out assertions passed, so an honest 80 percent implementation scores 0.80 rather than zero. A task only counts as passed at the full ship bar. Repository tasks additionally require the complete pre-existing suite to stay green.
Agreement
Grading is deterministic and execution-based, so there is no inter-rater statistic. Discrimination is evidenced instead by the known-bad and upstream gates, both of which are required to fail.
RL Training Environments
Each task is a reward-bearing environment: deterministic setup, a machine oracle, and a verified pass/fail signal. Run policy rollouts against them without writing the graders yourself.
Agentic Distillation
Train on verified-correct trajectories from a stronger teacher, rejection-sampled so only runs that actually passed the held-out suite survive.
Contamination-Safe Evaluation
Invented and mutated specifications have no public reference implementation to leak, which keeps the evaluation honest for models trained on the open web.
How you load it
Delivery
Secure download, Docker image, Git bundle
Formats
Dockerfile + pinned image, JSONL task index, Patch, pytest suite
Auth
Org-scoped credentials with per-customer delivery paths. Held-out graders ship under a separate restricted licence from the solver-visible specifications so an evaluation set can be split from a training set at delivery.
Cadence
Continuous build. Tranches are cut monthly; a month of production is on hand at the volumes shown above.
# Grade a candidate against one held-out environmentbash harness/grade_in_docker.sh \--task <task-id> \--install ./candidate # path-only; bare requirement strings are refused# The runner executes with the network namespace off:# docker run --network none ... pip install --no-index ...# Score = passed_assertions / min_passed PASS only at the full ship barbash harness/verify_all.sh # reference PASS / known-bad FAIL / upstream FAIL
Built against
Request a sample
Repo Build & Repair Tasks
Real records in the delivery format, with the schema and provenance that come with them. Evaluation access is restricted-scope and moves under a mutual NDA, so tell us what you are building and we will send the slice that fits.