Computer-Use Trajectories
Recordings of agents using a browser or desktop: a screenshot and the page structure at every step, the action taken, and a check of the end result.
- 96,000computer-use trajectories delivered
Sample
One step and a recorded runOne step of a computer-use run
What the agent could see.
A screenshot and the accessibility tree, both taken at the same moment.
- screenshot
- The app as it looked before the action
- a11y_tree
- Each element’s role, name and state
- step
- Where this step falls in the run
Each observation is paired with the action that follows.
What the agent did.
The recorded action and the element it acted on.
- action.type
- Click, type, navigate or another action
- target_role
- The kind of element it acted on
- grounding
- Where on screen the action landed
Every action points to something the agent could see.
What changed, and what passed.
The app’s new state, and a check of the final result when the task ends.
- observation_delta
- What changed after the action
- next observation
- What the agent sees next
- final_state_check
- Whether the task got done, checked at the end
The environment decides pass or fail once the task ends.
One step of a computer-use trajectory
Representative shape. Application content and coordinates from the real capture are withheld.
{
"step": 14,
"screenshot": "steps/014.png",
"a11y_tree": "steps/014.a11y.json",
"action": {
"type": "click",
"target_role": "button",
"grounding": {
"x": "<withheld>",
"y": "<withheld>"
}
},
"observation_delta": "<withheld>",
"final_state_check": {
"deterministic": true,
"evaluated_at_end_of_episode": true
}
}Watch an agent work through a task
A recorded run from our research tasks, step by step: what the agent meant to do, the tool it called, what came back, and the grade at the end.
Research reproduction audit
Captured run, de-identified
Pass
Audit a one-dimensional convection PINN reproduction packet. Recompute the evidence from samples and distinguish a valid reproduction from fabricated metrics, a boundary mismatch, an unsatisfied residual, missing curvature gain, or insufficient final accuracy.
01 / 06Task
Toolbash
- Intent
Inspect the benchmark packet
- Tool call
Inspect the provided repository and the public benchmark specification.
- Observed result
The task defines a one-dimensional convection PINN reproduction with a reference PDE, initial condition, periodic boundary, and analytical solution. The working tree contains the model, PDE, reproduction script, requirements, and rubric materials.
repository inspected benchmark specification located
Recorded state2 records
surface status repository inspected reference PDE located
- Intent
Check the packet format
- Tool call
Inspect the available reproduction packet and rubric files.
- Observed result
The packet format and rubric are present. The auditor can be written against the normative fields without relying on a precomputed result file.
packet format checked rubric materials located
Recorded state2 records
surface status packet fields checked rubric located
- Intent
Write the auditor
- Tool call
Write a normalized audit artifact that recomputes the packet metrics and returns a classification.
- Observed result
The audit artifact is written. Its public role is to recompute evidence from raw packet samples, rather than trust a reported score.
normalized audit artifact written
Recorded state2 records
artifact status audit.py written output contract recomputed metrics and classification
- Intent
Exercise failure paths
- Tool call
Run local checks covering valid, fabricated, boundary, residual, curvature-gain, and final-accuracy cases.
- Observed result
The local checks return the expected classifications for the valid reproduction and each named failure mode, including insufficient final accuracy.
failure-path checks completed outputs summarized
Recorded state2 records
check family status valid reproduction classified failure paths classified
- Intent
Submit the audit run
- Tool call
Submit the completed audit artifact for machine grading.
- Observed result
The captured run reports the auditor as complete and ready for the machine grade.
audit artifact submitted awaiting terminal grade
Recorded state2 records
surface status submission complete grader pending
- Intent
Terminal grade
- Tool call
Evaluate the submitted auditor against six held-out reproduction-audit checks.
- Observed result
The machine grader reports a passing result for all six held-out checks.
verdict: PASS held-out checks: 6 / 6 score: 1.0
Recorded state2 records
grade field value held-out checks 6 / 6 verdict Pass
Specifications
4 parts| Availability | 96,000 recorded trajectories |
|---|---|
| Formats | JSONL trajectories, Screenshots, Accessibility trees, Final-state checks |
| Built for | Computer-use models, GUI agents and multimodal post-training |
What ships
Per-step observations
The screenshot and accessibility tree available before each action.
Grounded actions
The recorded tool action, target role and coordinate grounding for the interaction.
Observed transitions
The change in application state after the action, kept in the original sequence.
Outcome checks
A final-state check against the environment, evaluated after the task is complete.
Request a sample
We send real records in the format you would receive, with the field list and where each record came from. Access for evaluation is limited and covered by a mutual NDA.
ProductComputer-Use Trajectories