REAL-WORK RL ENVIRONMENTS
Back-Office Work Tasks
Real de-identified business operations turned into RL instances with exact-value oracles - the accounting close as an agent task, graded to the cent.
Inspect a real sample
Representative records in the delivery format, ready to inspect before licensing the full dataset.
Instance manifest for a corrupted-close environment
Real structure from a shipped taskset. Ledger contents, fault specifics, and oracle values are withheld - these are graded instances.
{
"instance_id": "<withheld>",
"source": { "operator": "<opaque token>", "period": "<withheld>" },
"fault": { "kind": "<one of the mixed-fault classes>", "affected_voucher": "<held out from agent>" },
"oracle": { "expected_cents": "<withheld>", "tolerance": 0 },
"scores": ["reconciliation", "diagnosis"],
"deidentification": { "identities": "stable tokens", "business": "opaque token",
"amounts_dates_structure": "preserved", "leak_audit": "passed" },
"difficulty_gate": { "frontier_models_tested": 2, "pass_rate": "0%" }
}One step of a computer-use trajectory
Representative shape. Application content and coordinates from the real capture are withheld.
{
"step": 14,
"screenshot": "steps/014.png",
"a11y_tree": "steps/014.a11y.json",
"action": { "type": "click", "target_role": "button", "grounding": { "x": "<withheld>", "y": "<withheld>" } },
"observation_delta": "<withheld>",
"final_state_check": { "deterministic": true, "evaluated_at_end_of_episode": true }
}Record shape
Every field, its type, whether it can be null, and a representative value.
| Field | Type | Constraint | Description |
|---|---|---|---|
| instance_id | string | required | Identifier for one corrupted-ledger instance. |
| fault.kind | enum | required | The class of corruption introduced into the real de-identified ledger. |
| fault.affected_voucher | string | required | The record the agent has to identify. Held out from the agent, known to the oracle. |
| oracle.expected_cents | int · cents | required | Exact tie-out value. Grading is to the cent, not to a tolerance band. |
| ledger/ | directory | required | The de-identified operating data the instance is built from. Identities are tokenized; amounts, dates, and structure are real. |
| steps[].screenshot | image | nullable | Per-step capture for computer-use trajectories. |
| steps[].a11y_tree | JSON | nullable | Accessibility tree at the step, which is what makes an action reconstructable rather than merely visible. |
| steps[].action | JSON | nullable | The action taken, with coordinate grounding preserved. |
| final_state_check | JSON | required | Deterministic assertion over the end state of the environment. |
Controlled-Corruption Environments
A real de-identified ledger is corrupted in a known way. The agent has to find the fault, fix it, and tie out the close. The oracle knows the exact cent and the exact affected voucher, so grading is deterministic.
Computer-Use Trajectories
Real browser and desktop agent runs against real applications, recorded per step with screenshot, accessibility tree, action, and a final-state checker. Coordinate grounding is preserved.
Cross-Tool Reconstruction Tasks
Tasks whose evidence is split across tools, so the model has to reconstruct what happened rather than answer from a single document.
How it is built
- 01
Originate the operating data
The substrate is real operating data captured from inside businesses actually in motion, not scraped from the web and not synthesized. This is the layer that cannot be obtained any other way.
- 02
De-identify without hollowing out
Identities become stable tokens and the business becomes an opaque token. Amounts, dates, sequence, and structure are preserved, because those are the signal. The rule is to remove who it is, never what happens.
- 03
Verify zero leakage
Every instance passes a leak check before it can enter a tranche. Automated scans run first, and a human read is the catch layer, because the failure mode that matters is the one an automated check already approved.
- 04
Introduce a controlled corruption
A known fault is injected into the de-identified ledger. Because the fault is authored, the oracle knows both the exact tie-out value and the exact affected record, which is what makes the reward deterministic.
- 05
Gate on frontier difficulty
Instances are measured against frontier models before shipping. Ones that are trivially solved do not earn a place in the taskset.
- 06
Capture computer-use trajectories per step
Browser and desktop runs are recorded as a sequence of screenshot, accessibility tree, action, and coordinate grounding, closed by a final-state checker rather than a self-reported success flag.
How we validate
What each evaluation measures and how it is run. Where no benchmark is published, we show the methodology and say so.
Frontier difficulty calibration
Measures
Whether the all-or-nothing professional deliverables are actually hard.
Method
Each eval surface is run against two frontier models in its own environment and graded by the task own machine oracle, in the easiest possible setting where every piece of evidence is supplied in context.
Result
Real measured: 0% on the legal, finance, medical, and cross-tool computer-use deliverables for both frontier models tested. One wrong figure fails the task.
Exact-cent reconciliation
Measures
Whether the agent tied the close out correctly.
Method
The submitted close is compared to the oracle value at cent precision, with no tolerance band.
Result
Real measured: deterministic per instance across the reference taskset.
Affected-voucher diagnosis
Measures
Whether the agent found the right cause, not just the right number.
Method
The identified record is compared against the known injected fault, scored separately from the arithmetic result.
Result
Real measured: deterministic per instance. Scored apart from reconciliation so a lucky number does not read as a diagnosis.
Leak audit
Measures
Whether any identifying material survived de-identification.
Method
Layered automated scans across every shipped artifact, followed by a human read of the delivered bytes rather than of the build own belief about them.
Result
Real measured: zero-leak verified per tranche before release.
Ground truth
What correct means for this data, and how it is established.
Ground truth
The authored corruption. Because the fault is injected deliberately into real data, both the correct end value and the correct causal record are known exactly.
How it is established
Two independent scores: an exact-cent reconciliation of the close, and a diagnosis check against the affected record. Computer-use trajectories are closed by a deterministic final-state check on the environment rather than by a self-reported outcome.
Agreement
Deterministic against the authored fault, so no inter-rater statistic applies. Difficulty is evidenced by frontier-model calibration rather than asserted, and the harness ships so it can be re-run.
Agents That Do Office Work
Models trained on the public web have never seen how a company actually closes a month. These environments supply the missing substrate with a verifiable outcome attached.
Hard Evaluation That Still Discriminates
All-or-nothing professional deliverables where one wrong figure fails the task. Both frontier models tested score zero, so the benchmark has headroom left in it.
Process Reward Modeling
Per-step records with a known-correct end state make it possible to score the path, not only the answer.
How you load it
Delivery
Secure download, Docker image, VM snapshot
Formats
Docker image, JSONL trajectory, Parquet ledger, Screenshot + accessibility tree
Auth
Org-scoped credentials with per-customer delivery paths. Every artifact ships de-identified and zero-leak verified; source businesses are never identified and never identifiable.
Cadence
Continuous build. Tranches are cut monthly; a month of production is on hand at the volumes shown above. Each newly onboarded de-identified operator adds a corpus rather than an increment.
# Run one corrupted-close instance and score itbash harness/run.sh --instance <id> --agent ./candidate# Two independent scores are returned:# reconciliation : exact cents, no tolerance band# diagnosis : affected voucher identified, scored separatelybash harness/calibrate.sh --model <model-id> # re-measure difficulty yourself
Built against
Request a sample
Back-Office Work Tasks
Real records in the delivery format, with the schema and provenance that come with them. Evaluation access is restricted-scope and moves under a mutual NDA, so tell us what you are building and we will send the slice that fits.