GPU RESEARCH ENVIRONMENTS
ML Research Tasks
GPU-backed environments where the task is to run machine-learning research: win the competition, replicate the paper, or beat the base model.
Inspect a real sample
Representative records in the delivery format, ready to inspect before licensing the full dataset.
Environment manifest for a rubric-graded replication
Real structure. Paper identity, rubric contents, and grader internals are withheld - the rubric is the graded artifact.
{
"env_id": "<withheld>",
"kind": "replication",
"rubric": { "leaves": 1963, "source": "upstream-authored", "weights": "rubric/weights.json" },
"judge": { "kind": "automated", "measured_f1_against_rubric": 0.946 },
"trajectory": { "captured": true, "graded": true, "self_reported": false },
"gpu_protocol": "A10-class, agent-driven replication",
"ships": ["grader/", "rubric/", "trajectory/"]
}Record shape
Every field, its type, whether it can be null, and a representative value.
| Field | Type | Constraint | Description |
|---|---|---|---|
| env_id | string | required | Identifier for the research environment. |
| kind | enum | required | competition | replication | post-training. e.g. replication |
| rubric.leaves | int · criteria | nullable | Leaf criteria in the upstream-authored grading rubric. e.g. 1963 |
| rubric.weights | JSON | nullable | Per-node weighting used to roll leaf scores into a replication score. |
| grader/ | directory | required | The real grader for the environment, not a proxy metric. |
| trajectory/ | directory | required | A captured agent run that was actually graded, shipped as evidence rather than described. |
| gpu_protocol | string | required | The accelerator class and run protocol the environment expects. e.g. H100 post-training protocol |
| contamination_guard | JSON | nullable | Check that a post-training gain came from training rather than from recall of the benchmark. |
| base_weights | path | nullable | Real starting weights the agent has to improve on. |
Competition Environments
A competition directory, a real grader, and a captured agent run that was medal-verified rather than self-reported. Runs execute on consumer-to-midrange accelerators.
Paper Replication Environments
The real upstream leaf rubric with a weighted grader and an automated judge measured against it, plus a captured replication trajectory that was actually graded.
Post-Training Environments
The agent sources and curates its own data, trains, and has to beat the base model across a fixed suite of benchmarks. Ships with a contamination guard and real weights.
How it is built
- 01
Use the real grader, not a proxy
Competition environments ship the actual competition grader and replication environments ship the real upstream leaf rubric. A proxy metric would make the reward cheap to game and the result meaningless.
- 02
Weight the rubric
Leaf criteria roll up through a weighted tree so a multi-day replication produces a dense scoring surface instead of one subjective judgement at the end.
- 03
Measure the automated judge against the rubric
The judge that scores replication work is itself evaluated against the rubric rather than trusted, and its agreement is reported as a number.
- 04
Capture and grade a real trajectory
Every environment ships with an agent run that was actually executed and actually graded. Medal or score claims are verified against the grader, not self-reported by the run.
- 05
Guard post-training against recall
A post-training environment requires the agent to source data, train, and beat the base model across a fixed suite. A contamination guard separates a real gain from the model having seen the benchmark.
- 06
Run on a stated accelerator protocol
Each environment declares the accelerator class it expects, from consumer-to-midrange cards for competition runs up to datacenter parts for post-training, so results are reproducible rather than hardware-dependent folklore.
How we validate
What each evaluation measures and how it is run. Where no benchmark is published, we show the methodology and say so.
Judge agreement against the rubric
Measures
Whether the automated replication judge matches the real rubric.
Method
The judge is scored against the upstream-authored leaf rubric and reported as an F1 rather than assumed correct.
Result
Real measured: F1 0.946 against the 1,963-leaf rubric.
Trajectory verification
Measures
Whether the shipped example runs actually achieved what they claim.
Method
Each captured trajectory is re-scored by the environment own grader; competition runs are medal-verified rather than self-reported.
Result
Real measured: every shipped trajectory in the reference tranche is graded evidence.
Base-model improvement
Measures
Whether a post-training run genuinely beat its starting point.
Method
The trained model is compared to the base model across a fixed benchmark suite, with a contamination guard applied to the result.
Result
Methodology-stage per run. The comparison and the guard ship with the environment so a buyer can reproduce the check.
Ground truth
What correct means for this data, and how it is established.
Ground truth
The real competition grader, the real upstream leaf rubric, and the measured benchmark suite for post-training runs.
How it is established
Competition runs are scored by the competition own grader and medal-verified. Replications roll weighted leaf criteria into a score, with the automated judge itself measured against the rubric. Post-training runs are scored by benchmark delta against the base model, filtered through a contamination guard.
Agreement
The replication judge reports F1 0.946 against the upstream rubric. Competition and post-training scoring are execution-based and deterministic.
Self-Improving Agent Research
The clearest verifiable reward in ML research is a number that went up on a held-out benchmark. These environments supply that loop end to end.
Rubric-Graded Long-Horizon Work
A 1,963-leaf rubric turns a multi-day replication into a dense scoring surface instead of a single subjective judgement.
Contamination-Controlled Measurement
A guard on the post-training suite keeps a model from scoring by recall, which is the usual failure mode of benchmark-driven training.
How you load it
Delivery
Secure download, Docker image, GPU protocol
Formats
Docker image, JSONL trajectory, Model checkpoint, Rubric JSON
Auth
Org-scoped credentials. Rubrics and graders are licensed separately from the environments so an evaluation set can be withheld from a training set.
Cadence
Continuous build, gated by accelerator availability rather than by authoring throughput. Tranches are cut monthly at the volumes shown above.
# Run one replication environment and score itbash harness/run.sh --env <env-id> --agent ./candidate --gpu a10# Weighted leaf rubric -> replication score.# The automated judge is itself scored against the rubric (F1 0.946).bash harness/verify_trajectory.sh --env <env-id> # re-grade the shipped run
Built against
Request a sample
ML Research Tasks
Real records in the delivery format, with the schema and provenance that come with them. Evaluation access is restricted-scope and moves under a mutual NDA, so tell us what you are building and we will send the slice that fits.