Engineering Delivery Records
Commits, pull requests, issues and deploys from real engineering teams, linked together.
Sample
3 recordsMerged PR resolving a linked issue (TypeScript)
Representative shape, not real customer code. base_ref_oid / head_ref_oid mirror the evaluator real PR fields; issue_linked is true.
{
"event_id": "dw-04412",
"org_id": "org_01",
"repo": "org/service-api",
"event_type": "pull_request.merged",
"actor_id": "u_07",
"timestamp": "2026-02-04T09:13:55Z",
"commit_sha": "625933e987e4",
"pr_number": 318,
"base_ref_oid": "19608d910738",
"head_ref_oid": "625933e987e4",
"issue_id": "ISSUE-204",
"issue_linked": true,
"changed_files": ["src/webhook/retry.ts", "src/webhook/retry.test.ts"],
"ci_status": "passed",
"test_command": "npm test -- retry",
"release_tag": "v2.14.0"
}Issue opened (problem end of the trace)
Representative. The resolving PR above closes this issue, forming one issue-to-code trace.
{
"event_id": "dw-04388",
"org_id": "org_01",
"repo": "org/service-api",
"event_type": "issue.opened",
"actor_id": "u_11",
"timestamp": "2026-02-03T17:40:02Z",
"issue_id": "ISSUE-204",
"issue_linked": true,
"ci_status": null,
"release_tag": null
}Warehouse query event (data-agent context)
Representative. Data-warehouse activity normalized into the same SDLC graph for data and analytics agent training.
{
"event_id": "dw-09120",
"org_id": "org_01",
"repo": null,
"event_type": "warehouse.query_run",
"actor_id": "u_07",
"timestamp": "2026-02-05T11:02:18Z",
"warehouse_query": "SELECT account_id, SUM(amount) FROM invoices WHERE status='open' GROUP BY 1",
"issue_linked": false,
"ci_status": null
}Specifications
| Formats | JSON, Parquet |
|---|---|
| Delivery | S3, REST API, Webhook, Restricted agreement / data room |
| Cadence | Real-time via webhooks or daily batch |
| Coverage | Continuous |
| Tool coverage | Source control, issues, docs, warehouse |
| Event types | 50+ SDLC events normalized |
| Data model | Commit - PR - issue - deploy graph |
| Schema | Unified engineering activity graph |
| Access | Restricted access under a signed agreement; per-recipient export. Raw repositories stay gated; only consented, sanitized material is shared. Source identities are anonymous; full provenance under NDA. |
What ships
Source Control Activity
Commits, pull requests, reviews, merges, and release tags with full diff and review-thread context. Author and repository entity-resolved.
Issue & Project Graph
Tickets, sprints, cycle times, and dependency edges across issue trackers. Joined to source-control activity via issue-PR linkage.
Data Warehouse Activity
Query patterns, dashboard usage, pipeline runs, and model refresh activity across analytics platforms. Useful for data-agent training.
Fields
17 fields
| Field | Description |
|---|---|
| event_idstring | Stable id for one normalized SDLC event (commit, PR, issue, review, ci_run, release, warehouse_query).Example dw-04412 |
| org_idstring | Partner org boundary; entities are never merged across partners.Example org_01 |
| repostring · nullable | Repository the event belongs to (owner/name), entity-resolved.Example org/service-api |
| event_typestring | One of 50+ normalized SDLC event types.Example pull_request.merged |
| actor_idstring | Resolved author/actor node, unified across tools.Example u_07 |
| timestampstring · ISO-8601 | Event time in UTC for point-in-time reconstruction.Example 2026-02-04T09:13:55Z |
| commit_shastring · nullable | Commit hash for commit/merge events; anchors repo state.Example 625933e987e4 |
| pr_numberint · nullable | Pull request number for PR/review events (evaluator field).Example 318 |
| base_ref_oidstring · nullable | PR base commit SHA - state before the change (evaluator field baseRefOid).Example 19608d910738 |
| head_ref_oidstring · nullable | PR head commit SHA - the proposed change (evaluator field headRefOid).Example 625933e987e4 |
| issue_idstring · nullable | Linked issue joined to the resolving commits and PR.Example ISSUE-204 |
| issue_linkedbool | Whether this PR/commit is joined to a resolving issue (evaluator metric issue_linked_pr_ratio).Example true |
| changed_filesstring[] · nullable | Files touched by the change.Example ["src/webhook/retry.ts"] |
| ci_statusstring · nullable | Build/test signal at the event (CI run outcome).Example passed |
| test_commandstring · nullable | Recorded test command for execution-based verification.Example npm test -- retry |
| release_tagstring · nullable | Release that shipped the change.Example v2.14.0 |
| warehouse_querystring · sql · nullable | For data-warehouse events: query text for data-agent training.Example SELECT account_id, SUM(amount) ... |
Load it
The first commands after an approved delivery.
# SDLC point-in-time SWE task flow (mirrors the reference evaluator) # clone -> fetch repo from its git bundle # reconstruct base -> checkout base_ref_oid (state before the fix) # pose issue -> hand issue_title + issue_body to the agent-under-test # agent attempts -> agent reads/edits the repo to resolve the issue # run tests -> execute test_command (execution-based verification) # compare -> diff the agent change against the gold PR # verdict -> pass / fail + per-stage report # Trace quality is pre-graded by eval_kit (issue/patch/test clarity + alignment).
Quality checks
3 checksPoint-in-Time SWE Task
Whether an agent resolves a real issue the way a human engineer did.
Reconstruct base state from commit history and release tags; pose the linked issue while withholding the resolving commits and PR; score against the gold PR via execution-based verification (run the recorded test command) plus diff comparison.
Methodology-stage. The research is explicit that no benchmark numbers are published; quantifying agent performance is future work.
Task-Quality Rubric (pre-eval filtering)
Whether an issue-to-PR trace is clean enough to become an eval task.
A reference evaluator (eval_kit v0.6.2) scores each candidate PR on issue clarity, gold-patch clarity, test clarity, and patch/test/issue alignment, and tracks pass-first-filter rate, issue-linked-PR ratio, and PR acceptance rate. PRs without tests are rejected.
Methodology-stage. These are filtering and integrity rubrics run on real repositories, not a published agent-performance benchmark.
Data / Analytics Agent Task
Whether an agent can write SQL, fix pipelines, and answer analytical questions grounded in real warehouse activity.
Replay query patterns, dashboard usage, pipeline runs, and model-refresh activity from production warehouses as grounded context.
Methodology-stage. No measured outcome is reported.
Ground truth
| What is correct | The human pull request that actually merged and resolved the linked issue - the change that passed the repository own review and CI - together with the repository existing test suite at that point in time. |
|---|---|
| How it is established | Execution-based verification plus diff comparison against the gold PR. Trace quality is graded by a reference evaluator (eval_kit v0.6.2) on per-PR rubrics and per-repo integrity metrics: issue_linked_pr_ratio, commits_referencing_issues, pr_acceptance_rate, has_ci_cd, and test-framework presence. The harness has been run end to end on real repositories. |
| Agreement | Correctness is anchored to the repository own tests and the merged human change, not a separate human-rater pass. No inter-rater agreement figure is published at this stage. |
How it is built
- 01
Consented capture across SDLC tools
Founder-owned and partner repositories and their issue trackers are captured with consent. Source control contributes commits with full per-commit diff history plus PRs, reviews, linked issues, CI runs, and release tags.
- 02
Normalization to a unified engineering graph
Tool-specific records from source control, issue trackers, and warehouses are normalized into one SDLC event vocabulary of 50+ types, with author and repository entity resolution.
- 03
Issue-to-code linkage
Tickets are joined to the specific commits and PRs that resolved them, turning each resolved issue into a six-stage trace: problem, change, proposal, scrutiny, verification, outcome.
- 04
Quality filtering
Traces without a clean issue-to-PR join are not promoted to task records, and PRs are filtered against minimum criteria (for example test-file presence) before they become eval-grade.
- 05
Sanitization
Material is processed to remove sensitive content before inclusion; credentialed remotes and secrets are stripped.
- 06
Point-in-time reconstruction
Commit history and release tags recover the repository exactly as it stood just before the resolving change, so an agent sees only what was available then - no look-ahead into the fix.
Request a sample
We send real records in the format you would receive, with the field list and where each record came from. Access for evaluation is limited and covered by a mutual NDA.
ProductEngineering Delivery Records