Real Business Environments
Real company systems and records, with names removed, set up for an agent to work in. Experts write the tasks, and the grading stays hidden from the agent.
Sample
An example workflow and a recorded runA service-to-billing workflow
Linked records from one company, names removed.
The case, the authorization, the service record and the accounting entry share a history. The links between them stay intact.
- Case record
- Request, status and prior history
- Authorization
- Coverage, approval and limits
- Service record
- What was delivered and when
- Claim and ledger
- Billing, posting and reconciliation
Records stay linked across systems and over time.
Find and fix the service-billing mismatch.
The agent gets a clear business goal and access to the systems it needs to look into the problem and fix it.
- Task
- Find and fix the mismatch between service and billing
- Evidence
- Case history, messages and policy
- Actions
- Read records, check the mismatch, update the workflow
- Limits
- The task’s permissions and business rules
Graded on the records, not on the agent’s report.
The final state is checked against expert answers, with checks the agent never sees.
- Outcome
- The affected records reconcile
- Diagnosis
- The real cause is found
- Consistency
- Linked records agree afterwards
- Baseline
- Pass rate measured against compute spent
Hidden checks and baseline runs show how hard each task is.
Watch an agent work through a task
A recorded run from our research tasks, step by step: what the agent meant to do, the tool it called, what came back, and the grade at the end.
Research reproduction audit
Captured run, de-identified
Pass
Audit a one-dimensional convection PINN reproduction packet. Recompute the evidence from samples and distinguish a valid reproduction from fabricated metrics, a boundary mismatch, an unsatisfied residual, missing curvature gain, or insufficient final accuracy.
01 / 06Task
Toolbash
- Intent
Inspect the benchmark packet
- Tool call
Inspect the provided repository and the public benchmark specification.
- Observed result
The task defines a one-dimensional convection PINN reproduction with a reference PDE, initial condition, periodic boundary, and analytical solution. The working tree contains the model, PDE, reproduction script, requirements, and rubric materials.
repository inspected benchmark specification located
Recorded state2 records
surface status repository inspected reference PDE located
- Intent
Check the packet format
- Tool call
Inspect the available reproduction packet and rubric files.
- Observed result
The packet format and rubric are present. The auditor can be written against the normative fields without relying on a precomputed result file.
packet format checked rubric materials located
Recorded state2 records
surface status packet fields checked rubric located
- Intent
Write the auditor
- Tool call
Write a normalized audit artifact that recomputes the packet metrics and returns a classification.
- Observed result
The audit artifact is written. Its public role is to recompute evidence from raw packet samples, rather than trust a reported score.
normalized audit artifact written
Recorded state2 records
artifact status audit.py written output contract recomputed metrics and classification
- Intent
Exercise failure paths
- Tool call
Run local checks covering valid, fabricated, boundary, residual, curvature-gain, and final-accuracy cases.
- Observed result
The local checks return the expected classifications for the valid reproduction and each named failure mode, including insufficient final accuracy.
failure-path checks completed outputs summarized
Recorded state2 records
check family status valid reproduction classified failure paths classified
- Intent
Submit the audit run
- Tool call
Submit the completed audit artifact for machine grading.
- Observed result
The captured run reports the auditor as complete and ready for the machine grade.
audit artifact submitted awaiting terminal grade
Recorded state2 records
surface status submission complete grader pending
- Intent
Terminal grade
- Tool call
Evaluate the submitted auditor against six held-out reproduction-audit checks.
- Observed result
The machine grader reports a passing result for all six held-out checks.
verdict: PASS held-out checks: 6 / 6 score: 1.0
Recorded state2 records
grade field value held-out checks 6 / 6 verdict Pass
Specifications
4 parts| Availability | Real company source material · Environments configured to your task |
|---|---|
| Formats | Connected source records, Runnable tool surface, Private grader, Baseline rollouts |
| Built for | Business agents, long-horizon RL and workflow evaluation |
What ships
Connected starting state
Company records, communication, actors, policies and business-system state, linked across the workflow.
A task and its tools
An expert-defined objective with the tools, constraints and environment state the agent needs to act.
Ground truth and a private grader
Expected outcomes and deterministic checks kept separate from the agent-visible task.
Measured baseline runs
Captured rollouts, failure cases and pass-rate-versus-compute curves for the scoped task family.
Request a sample
We send real records in the format you would receive, with the field list and where each record came from. Access for evaluation is limited and covered by a mutual NDA.
ProductReal Business Environments