Agent environments & trajectories
Business and research tasks that run as delivered, with recorded agent runs: what the agent saw, what it did and how it was graded.
- 64,000graded back-office tasks delivered
- 96,000computer-use trajectories delivered
Sample
Recorded runs, names removedWatch an agent work through a task
A recorded run from our research tasks, step by step: what the agent meant to do, the tool it called, what came back, and the grade at the end.
Research reproduction audit
Captured run, de-identified
Pass
Audit a one-dimensional convection PINN reproduction packet. Recompute the evidence from samples and distinguish a valid reproduction from fabricated metrics, a boundary mismatch, an unsatisfied residual, missing curvature gain, or insufficient final accuracy.
01 / 06Task
Toolbash
- Intent
Inspect the benchmark packet
- Tool call
Inspect the provided repository and the public benchmark specification.
- Observed result
The task defines a one-dimensional convection PINN reproduction with a reference PDE, initial condition, periodic boundary, and analytical solution. The working tree contains the model, PDE, reproduction script, requirements, and rubric materials.
repository inspected benchmark specification located
Recorded state2 records
surface status repository inspected reference PDE located
- Intent
Check the packet format
- Tool call
Inspect the available reproduction packet and rubric files.
- Observed result
The packet format and rubric are present. The auditor can be written against the normative fields without relying on a precomputed result file.
packet format checked rubric materials located
Recorded state2 records
surface status packet fields checked rubric located
- Intent
Write the auditor
- Tool call
Write a normalized audit artifact that recomputes the packet metrics and returns a classification.
- Observed result
The audit artifact is written. Its public role is to recompute evidence from raw packet samples, rather than trust a reported score.
normalized audit artifact written
Recorded state2 records
artifact status audit.py written output contract recomputed metrics and classification
- Intent
Exercise failure paths
- Tool call
Run local checks covering valid, fabricated, boundary, residual, curvature-gain, and final-accuracy cases.
- Observed result
The local checks return the expected classifications for the valid reproduction and each named failure mode, including insufficient final accuracy.
failure-path checks completed outputs summarized
Recorded state2 records
check family status valid reproduction classified failure paths classified
- Intent
Submit the audit run
- Tool call
Submit the completed audit artifact for machine grading.
- Observed result
The captured run reports the auditor as complete and ready for the machine grade.
audit artifact submitted awaiting terminal grade
Recorded state2 records
surface status submission complete grader pending
- Intent
Terminal grade
- Tool call
Evaluate the submitted auditor against six held-out reproduction-audit checks.
- Observed result
The machine grader reports a passing result for all six held-out checks.
verdict: PASS held-out checks: 6 / 6 score: 1.0
Recorded state2 records
grade field value held-out checks 6 / 6 verdict Pass
Instance manifest for a corrupted-close environment
Real structure from a shipped taskset. Ledger contents, fault specifics, and oracle values are withheld - these are graded instances.
{
"instance_id": "<withheld>",
"source": {
"operator": "<opaque token>",
"period": "<withheld>"
},
"fault": {
"kind": "<one of the mixed-fault classes>",
"affected_voucher": "<held out from agent>"
},
"oracle": {
"expected_cents": "<withheld>",
"tolerance": 0
},
"scores": ["reconciliation", "diagnosis"],
"deidentification": {
"identities": "stable tokens",
"business": "opaque token",
"amounts_dates_structure": "preserved",
"leak_audit": "passed"
},
"difficulty_gate": {
"frontier_models_tested": 2,
"pass_rate": "0%"
}
}Specifications
4 parts| Availability | Existing environment catalogue, plus tailored variants |
|---|---|
| Formats | Docker + VM, JSONL + Parquet, Screenshots + a11y, Rubric JSON |
| Built for | Agentic RL, tool use, evaluation & post-training |
| Used for | Agentic reinforcement learning, Supervised post-training, Trajectory distillation, Held-out evaluation |
Available now
| Business environments | 64,000 graded instances |
|---|---|
| Computer use | 96,000 trajectories |
| ML competitions | 1,600 runs |
| Paper replication | 800 replications |
| Post-training | 960 runs |
What ships
Runnable agent environments
The task, source material, tools, starting state, constraints, and resettable execution surface in one package.
Complete trajectories
Tool calls, screenshots, accessibility trees, observations, state transitions, checkpoints, and the finished result.
Rewards and evaluation
Exact-value oracles, final-state checks, benchmark scores, or dense rubrics matched to the kind of work the agent performs.
Training-ready task families
Existing environment families plus new variants across domain, tool surface, context length, difficulty, and reward shape.
Quality
Method notePartial Credit Is What Turns a Benchmark Into an Environment
A plain pass or fail hides how close an agent got. Our note shows how partial scores make that progress visible, and how to test the grader itself.
Request a sample
We send real records in the format you would receive, with the field list and where each record came from. Access for evaluation is limited and covered by a mutual NDA.
ProductAgent environments & trajectories