Skip to content
gerra
Let’s talk

Back-Office Work Tasks

Real de-identified business operations turned into RL instances with exact-value oracles - the accounting close as an agent task, graded to the cent.

Request a sample
16,000 RL INSTANCES · 24K TRAJECTORIESDocker · JSONL · Parquet · Screenshot + a11y treeContinuous build; monthly tranches
16K
Graded RL instances
24K
Computer-use trajectories
0%
Frontier-model pass rate
Exact
Cent-level oracle
01Download

Inspect a real sample

Representative records in the delivery format, ready to inspect before licensing the full dataset.

Instance manifest for a corrupted-close environment

Real structure from a shipped taskset. Ledger contents, fault specifics, and oracle values are withheld - these are graded instances.

instance.jsonrepresentative
{
  "instance_id": "<withheld>",
  "source": { "operator": "<opaque token>", "period": "<withheld>" },
  "fault": { "kind": "<one of the mixed-fault classes>", "affected_voucher": "<held out from agent>" },
  "oracle": { "expected_cents": "<withheld>", "tolerance": 0 },
  "scores": ["reconciliation", "diagnosis"],
  "deidentification": { "identities": "stable tokens", "business": "opaque token",
                        "amounts_dates_structure": "preserved", "leak_audit": "passed" },
  "difficulty_gate": { "frontier_models_tested": 2, "pass_rate": "0%" }
}

One step of a computer-use trajectory

Representative shape. Application content and coordinates from the real capture are withheld.

trajectory.jsonlrepresentative
{
  "step": 14,
  "screenshot": "steps/014.png",
  "a11y_tree": "steps/014.a11y.json",
  "action": { "type": "click", "target_role": "button", "grounding": { "x": "<withheld>", "y": "<withheld>" } },
  "observation_delta": "<withheld>",
  "final_state_check": { "deterministic": true, "evaluated_at_end_of_episode": true }
}
02Schema

Record shape

Every field, its type, whether it can be null, and a representative value.

View the record schema 9 fields

instance_id

required

Identifier for one corrupted-ledger instance.

Type
string

fault.kind

required

The class of corruption introduced into the real de-identified ledger.

Type
enum

fault.affected_voucher

required

The record the agent has to identify. Held out from the agent, known to the oracle.

Type
string

oracle.expected_cents

required

Exact tie-out value. Grading is to the cent, not to a tolerance band.

Type
int
Unit
cents

ledger/

required

The de-identified operating data the instance is built from. Identities are tokenized; amounts, dates, and structure are real.

Type
directory

steps[].screenshot

nullable

Per-step capture for computer-use trajectories.

Type
image

steps[].a11y_tree

nullable

Accessibility tree at the step, which is what makes an action reconstructable rather than merely visible.

Type
JSON

steps[].action

nullable

The action taken, with coordinate grounding preserved.

Type
JSON

final_state_check

required

Deterministic assertion over the end state of the environment.

Type
JSON
03What's included

Controlled-Corruption Environments

A real de-identified ledger is corrupted in a known way. The agent has to find the fault, fix it, and tie out the close. The oracle knows the exact cent and the exact affected voucher, so grading is deterministic.

Computer-Use Trajectories

Real browser and desktop agent runs against real applications, recorded per step with screenshot, accessibility tree, action, and a final-state checker. Coordinate grounding is preserved.

Cross-Tool Reconstruction Tasks

Tasks whose evidence is split across tools, so the model has to reconstruct what happened rather than answer from a single document.

04Methodology

How it is built

  1. 01

    Originate the operating data

    The substrate is real operating data captured from inside businesses actually in motion, not scraped from the web and not synthesized. This is the layer that cannot be obtained any other way.

  2. 02

    De-identify without hollowing out

    Identities become stable tokens and the business becomes an opaque token. Amounts, dates, sequence, and structure are preserved, because those are the signal. The rule is to remove who it is, never what happens.

  3. 03

    Verify zero leakage

    Every instance passes a leak check before it can enter a tranche. Automated scans run first, and a human read is the catch layer, because the failure mode that matters is the one an automated check already approved.

  4. 04

    Introduce a controlled corruption

    A known fault is injected into the de-identified ledger. Because the fault is authored, the oracle knows both the exact tie-out value and the exact affected record, which is what makes the reward deterministic.

  5. 05

    Gate on frontier difficulty

    Instances are measured against frontier models before shipping. Ones that are trivially solved do not earn a place in the taskset.

  6. 06

    Capture computer-use trajectories per step

    Browser and desktop runs are recorded as a sequence of screenshot, accessibility tree, action, and coordinate grounding, closed by a final-state checker rather than a self-reported success flag.

05Evals

How we validate

What each evaluation measures and how it is run. Where no benchmark is published, we show the methodology and say so.

Frontier difficulty calibration

Measures

Whether the all-or-nothing professional deliverables are actually hard.

Method

Each eval surface is run against two frontier models in its own environment and graded by the task own machine oracle, in the easiest possible setting where every piece of evidence is supplied in context.

Result

Real measured: 0% on the legal, finance, medical, and cross-tool computer-use deliverables for both frontier models tested. One wrong figure fails the task.

Exact-cent reconciliation

Measures

Whether the agent tied the close out correctly.

Method

The submitted close is compared to the oracle value at cent precision, with no tolerance band.

Result

Real measured: deterministic per instance across the reference taskset.

Affected-voucher diagnosis

Measures

Whether the agent found the right cause, not just the right number.

Method

The identified record is compared against the known injected fault, scored separately from the arithmetic result.

Result

Real measured: deterministic per instance. Scored apart from reconciliation so a lucky number does not read as a diagnosis.

Leak audit

Measures

Whether any identifying material survived de-identification.

Method

Layered automated scans across every shipped artifact, followed by a human read of the delivered bytes rather than of the build own belief about them.

Result

Real measured: zero-leak verified per tranche before release.

06Graders

Ground truth

What correct means for this data, and how it is established.

Ground truth

The authored corruption. Because the fault is injected deliberately into real data, both the correct end value and the correct causal record are known exactly.

How it is established

Two independent scores: an exact-cent reconciliation of the close, and a diagnosis check against the affected record. Computer-use trajectories are closed by a deterministic final-state check on the environment rather than by a self-reported outcome.

Agreement

Deterministic against the authored fault, so no inter-rater statistic applies. Difficulty is evidenced by frontier-model calibration rather than asserted, and the harness ships so it can be re-run.

07Application

Agents That Do Office Work

Models trained on the public web have never seen how a company actually closes a month. These environments supply the missing substrate with a verifiable outcome attached.

Hard Evaluation That Still Discriminates

All-or-nothing professional deliverables where one wrong figure fails the task. Both frontier models tested score zero, so the benchmark has headroom left in it.

Process Reward Modeling

Per-step records with a known-correct end state make it possible to score the path, not only the answer.

08Environment & integration

How you load it

Delivery

Secure download, Docker image, VM snapshot

Formats

Docker image, JSONL trajectory, Parquet ledger, Screenshot + accessibility tree

Auth

Org-scoped credentials with per-customer delivery paths. Every artifact ships de-identified and zero-leak verified; source businesses are never identified and never identifiable.

Cadence

Continuous build. Tranches are cut monthly; a month of production is on hand at the volumes shown above. Each newly onboarded de-identified operator adds a corpus rather than an increment.

quickstart.sh
# Run one corrupted-close instance and score it
bash harness/run.sh --instance <id> --agent ./candidate
 
# Two independent scores are returned:
#   reconciliation : exact cents, no tolerance band
#   diagnosis      : affected voucher identified, scored separately
 
bash harness/calibrate.sh --model <model-id>   # re-measure difficulty yourself

Built against

GDPval (OpenAI)HealthBenchFinanceBenchBigFinanceBenchLegalBenchCUADOSWorld

Request a sample

Back-Office Work Tasks

Real records in the delivery format, with the schema and provenance that come with them. Evaluation access is restricted-scope and moves under a mutual NDA, so tell us what you are building and we will send the slice that fits.

or email team@gerra.com