Git history contains a natural record of software problems and human responses. A repository before a change defines the available context. The commit or pull request describes what changed. Tests and later behavior can provide evidence about whether the change worked.
That makes version history a promising substrate for coding-agent evaluation. It does not make every commit a benchmark.
The hard part is constructing a task that is point-in-time, reproducible, and scored against evidence the agent could not see in advance.
Start with a real historical boundary
A sound task needs at least three revisions:
- Base revision: the repository state presented to the agent.
- Reference change: the authentic human patch used as evidence, not necessarily the only acceptable answer.
- Verification revision: the tests or assertions used to judge behavior.
The base revision must not contain the answer. Retrieval indexes, documentation, issue text, and dependency metadata also need a timestamp boundary. Otherwise an evaluation can accidentally expose later code or language that tells the agent exactly what to do.
This is one reason cloneable history matters. A flat before-and-after file pair does not preserve the full tree, parent relationships, or the surrounding changes needed to audit leakage.
Not every commit is a useful task
Many commits are poor evaluation candidates:
- formatting-only changes;
- dependency bumps with generated lockfile noise;
- merges that combine unrelated work;
- large mechanical renames;
- changes that depend on private services or unavailable data;
- patches whose tests already pass at the base revision; and
- changes whose expected behavior cannot be stated independently of the patch.
A high commit count therefore says little about benchmark yield on its own. Useful task mining requires filtering, replay, and human review.
A six-step construction method
1. Select an independently understandable change
Prefer a focused bug fix, feature adjustment, regression repair, or refactor with observable behavior. Keep multi-file changes when the cross-file dependency is the point; do not force every task into a single-file template.
2. Reconstruct the base environment
Check out the exact parent revision. Record the runtime, package manager, lockfile, system dependencies, and setup commands. If the environment cannot be reproduced, label the task as static or partial rather than claiming execution-based grading.
3. Separate problem evidence from solution evidence
The task description may come from an issue, failing test, error report, or behavior specification. The reference patch, later commit messages, and post-fix documentation stay hidden from the agent.
4. Build discriminating verification
The strongest check fails on the base revision and passes after an acceptable fix. Existing tests are useful, but task-specific tests often need to be isolated from the later revision. Always run both sides. A test that passes before the patch proves nothing about the target change.
5. Permit equivalent solutions
Exact patch matching is usually too narrow. An agent can produce a behaviorally correct solution with different code. Score observable behavior first, then consider secondary properties such as scope, regressions, style, or unnecessary changes.
6. Preserve the audit trail
Store repository ID, base hash, hidden verification hash, environment manifest, test output, reference patch, selection reason, and exclusion notes. Bind these artifacts with checksums so the evaluation can be reproduced later.
What SWE-bench established
SWE-bench demonstrated a widely used pattern: pair real GitHub issues with corresponding pull requests, install the repository at the base commit, and evaluate candidate patches with tests. Its official benchmark documentation also makes clear how much engineering sits behind the task format: environment construction, version pinning, test specification, and contamination controls.
The lesson is broader than one benchmark. Historical authenticity is valuable only when the task can be replayed and the grader distinguishes failure from success.
Where hidden leakage enters
Leakage can appear through more than the target patch:
- a later test name describes the fix;
- a generated index includes post-base files;
- documentation was updated after the change;
- a package version points to a repaired implementation;
- commit messages or branch names reveal the solution; or
- the same repository appears in training data under another mirror.
Exact and near-duplicate detection should therefore operate across repositories and across time. Point-in-time retrieval must index only the base tree. Contamination risk should be reported, not waved away.
How a full-history corpus supports evaluation work
The Gerra Codebase Collection retains commit topology, timestamps, technical messages, branches, tags, merges, current snapshots, and cloneable repository bundles across 178 codebases. The release includes 15,238 unique commit objects and 61,454 unique reachable historical text blobs.
Those objects are raw material for evaluation construction. They are not advertised as 15,238 ready-made benchmarks. Turning history into a trustworthy task still requires the selection, environment, replay, and grading work described above.
The downloadable full-history repository sample contains 23 retained commits and five authentic patch pairs. Verify the archive, clone its bundle, and inspect all retained refs before deciding whether the history supports the tasks you want to construct. It does not make a runtime test-pass claim because the original lockfile and exact Hardhat dependency tree are unavailable.
A practical acceptance checklist
Before admitting a historical task to an evaluation set, require clear answers to these questions:
- Is the base revision exact and cloneable?
- Is the problem understandable without seeing the reference patch?
- Does the verifier fail before and pass after a valid change?
- Are dependencies and external services reproducible?
- Has post-base context been excluded from retrieval?
- Can equivalent correct patches pass?
- Are duplicates and contamination risks measured?
- Are every input, revision, and result bound to an audit record?
Git history is valuable because it records real evolution. A credible evaluation preserves that history, then adds a strict point-in-time boundary and an executable definition of success.