END-TO-END BUILD ENVIRONMENTS
Full-App Build Tasks
Non-technical prose specifications for whole web applications, graded by a browser driving the running app rather than by reading the diff.
Inspect a real sample
Representative records in the delivery format, ready to inspect before licensing the full dataset.
Substep tree for one application specification
Real structure with the specification withheld. The prose requirement is the graded artifact and is not published.
{
"spec_id": "<withheld>",
"grading_tier": "browser",
"substep_tree": [
{ "id": "1", "check": "application serves and reaches a usable initial state" },
{ "id": "1.1", "check": "seeded records visible", "depends_on": "1" },
{ "id": "2", "check": "primary create workflow completes end to end" },
{ "id": "2.1", "check": "created record survives reload", "depends_on": "2" },
{ "id": "3", "check": "constraint violation is rejected and surfaced to the user" }
],
"harness_selfcheck": {
"reference": "all-green required",
"stub": "partial required, dependents SKIPped"
}
}Record shape
Every field, its type, whether it can be null, and a representative value.
| Field | Type | Constraint | Description |
|---|---|---|---|
| spec_id | string | required | Identifier for the application specification. |
| spec.md | Markdown | required | The requirement written as non-technical prose, the only file the solver receives. |
| substep_tree | object | required | Ordered tree of user-visible behaviors, each independently checkable, producing dense partial credit. |
| grading_tier | enum | required | http | browser. Determines whether assertions run at the API or through a driven browser. e.g. browser |
| sandbox | directory | required | Seeded starting state so every run begins from identical data. |
| reference/ | directory | required | A complete implementation required to score all-green. |
| stub/ | directory | required | A deliberately partial implementation required to score partial with SKIP propagation. |
| defect | string | nullable | For spec-to-app tasks, the injected defect drawn from the stack-by-defect matrix. |
Prose-Spec App Builds
The specification is written the way a non-technical customer would write it. The grader walks a substep tree against the deployed application, so partial progress scores instead of collapsing to zero.
Stack x Defect Matrix
Spec-to-app tasks generated across a matrix of stacks and injected defects, each shipped as a seeded sandbox with deterministic end-to-end assertions.
Live-Deployment Grading
Two tiers: HTTP-level assertions for API behavior and a real browser for user-visible workflows. Grading runs against the running application, not against source inspection.
How it is built
- 01
Write the specification as a customer would
Requirements are authored in non-technical prose rather than as an engineering ticket, so the task measures whether a model can resolve ambiguity into a working product rather than follow a precise order.
- 02
Decompose into a substep tree
Each specification is broken into an ordered tree of user-visible behaviors. Reward is dense because a run that delivers half the application scores half rather than failing outright.
- 03
Seed the sandbox
Every task starts from identical seeded data, which is what makes two runs comparable and makes a deterministic assertion possible at all.
- 04
Grade the running application
Assertions execute against the deployed application - HTTP-level for API behavior, a real driven browser for user-visible workflows. Nothing is scored by reading the source.
- 05
Validate the harness itself
Before release, the reference implementation must score all-green and the deliberately partial stub must score partial with skips propagating correctly. A scoring change then reflects the model rather than a broken grader.
- 06
Generate the stack-by-defect matrix
Spec-to-app tasks are produced across a matrix of technology stacks and injected defects, so coverage is systematic rather than whatever happened to be convenient to author.
How we validate
What each evaluation measures and how it is run. Where no benchmark is published, we show the methodology and say so.
Reference all-green
Measures
Whether a correct implementation actually scores full marks.
Method
The authored reference is run through the complete substep tree and required to pass every node.
Result
Real measured: every reference in the shipped tranche scores all-green.
Stub partial with SKIP propagation
Measures
Whether partial credit and dependent-node skipping behave correctly.
Method
A deliberately incomplete implementation is required to score partial, with nodes that depend on missing behavior marked skipped rather than failed.
Result
Real measured: every stub in the shipped tranche scores partial with skips propagating as designed.
Model difficulty
Measures
How far a model gets through the substep tree.
Method
The shipped harness runs any model end to end and reports the fraction of the tree satisfied against the deployed application.
Result
Methodology-stage. The harness ships so difficulty is re-measurable per buyer and per model.
Ground truth
What correct means for this data, and how it is established.
Ground truth
The substep tree, anchored by a reference implementation that must satisfy every node and a stub that must not.
How it is established
Two tiers. HTTP assertions cover API contracts; a driven browser covers user-visible workflows against the live deployment. Score is the fraction of the substep tree satisfied, with dependent nodes skipped rather than failed when a prerequisite is missing.
Agreement
Deterministic against a seeded sandbox. Harness correctness is evidenced by the paired reference-passes and stub-partials checks rather than by inter-rater agreement.
Long-Horizon Agent RL
A whole application is a long-horizon task with dense intermediate reward, which is exactly what substep trees provide and single-assertion benchmarks do not.
Product-Sense Evaluation
Prose written by a non-engineer tests whether a model can resolve ambiguity into a working product, not whether it can follow a precise technical order.
Regression Harness
Reference-all-green and stub-partial checks mean the harness itself is validated, so a scoring change reflects the model rather than a broken grader.
How you load it
Delivery
Secure download, Docker image, Seeded sandbox
Formats
Dockerfile + pinned image, Markdown specification, JSONL substep tree, Browser automation script
Auth
Org-scoped credentials. Substep trees and reference implementations are licensed separately from the specifications so evaluation and training splits stay clean.
Cadence
Continuous build. Tranches are cut monthly; a month of production is on hand at the volumes shown above.
# Deploy a candidate and grade it through the browser tierbash harness/deploy.sh --spec <spec-id> --app ./candidatebash harness/grade.sh --spec <spec-id> --tier browser# Score = satisfied substep nodes / total, with dependent nodes SKIPped.# Harness self-check: reference must be all-green, stub must be partial.
Built against
Request a sample
Full-App Build Tasks
Real records in the delivery format, with the schema and provenance that come with them. Evaluation access is restricted-scope and moves under a mutual NDA, so tell us what you are building and we will send the slice that fits.