The easiest way to make a code dataset look impressive is to publish one enormous number. Tokens, files, lines, or repositories can all create the appearance of scale while hiding duplicates, generated material, shallow snapshots, broken history, or uncertain rights.
A buyer needs a different question: what exactly is the training or evaluation unit, and can the supplier prove that the delivered bytes match the description?
This checklist is designed for AI labs, evaluation teams, and data buyers reviewing repository-level code.
1. Define the unit of data
Ask whether each item is an isolated file, a current repository snapshot, a repository with history, or a constructed issue-and-patch task. Those units support different uses.
For repository-level work, request:
- a complete source tree at a named revision;
- stable repository and revision identifiers;
- imports, tests, configuration, and documentation;
- retained branches and tags, if history is part of the product; and
- an explicit list of material excluded from delivery.
Do not infer completeness from a ZIP that happens to contain many files.
2. Reproduce the headline counts
Every public metric should have a definition and a machine-readable source. “Lines of code” may mean all lines, nonblank lines, or language-classified code lines. “Commits” may be summed per repository or deduplicated by object identity. These are not interchangeable.
A good release distinguishes:
- current files from historical blobs;
- summed commits from unique commit objects;
- source from tests, documentation, generated, and vendored material;
- compressed archive size from extracted bytes; and
- exact characters or lines from estimated page equivalents.
Request the counting script or enough manifest detail to reproduce the result from a fresh extraction.
3. Verify Git integrity
If history is advertised, a directory of patches is not enough. The buyer should be able to clone or restore the retained graph.
For Git bundles, basic checks include:
git bundle verify repository.bundle
git clone repository.bundle audit-repo
git -C audit-repo fsck --strict
git -C audit-repo show-ref
git -C audit-repo log --all --graph --oneline
Compare bundle refs and object hashes with the release manifest. Check that commit parents, timestamps, messages, merges, branches, and tags match the supplier's preservation policy. If sanitization rewrote history, require a documented reconciliation process rather than an assurance that the repositories “still work.”
4. Measure duplicates and low-signal material
Exact duplicate files are only the first layer. The same project can appear under multiple mirrors, forks, renamed repositories, or slightly changed generated output.
Ask for:
- exact and near-duplicate methods;
- duplicate groups and the rule used to retain one copy;
- vendored dependency and generated-file measurements;
- fork handling; and
- the count before and after deduplication.
The StarCoder2 and The Stack v2 paper and The Stack v2 dataset card are useful public references for code-corpus governance and deduplication.
5. Treat tests as evidence, not decoration
Count test files, but do not stop there. A repository may contain stale, disabled, or non-discriminating tests.
For a sample, ask which checks were actually run in an isolated environment. Separate:
- syntax parsing;
- import or dependency resolution;
- build success;
- unit, integration, and end-to-end execution; and
- a demonstrated fail-before/pass-after task.
If dependencies or external services are unavailable, the supplier should say so. “Not executed” is more useful than an unsupported green checkmark.
6. Inspect privacy and secret controls
Private code can carry credentials and personal or customer information in current files and deep history. Scanning only the latest tree is insufficient.
Review the policy for:
- API keys, tokens, private keys, passwords, and connection strings;
- contributor email and sensitive contact details;
- customer or user records;
- private messages, paths, remotes, and source identifiers;
- binary and archive handling; and
- post-redaction re-scanning of every reachable object.
Ask whether findings were removed narrowly or whether entire repositories were silently dropped. Both approaches affect corpus composition and should be disclosed.
7. Review provenance and usage rights separately from technical quality
Technical cleanliness does not establish permission to use the data. Request source, ownership or licensing basis, contributor treatment, third-party components, intended permitted uses, transfer restrictions, and opt-out or deletion procedures.
Keep this diligence separate from code quality. A well-engineered repository can have unresolved rights. A clearly licensed repository can still be low-value training data. The buyer needs both answers.
8. Demand an inspectable sample
A useful sample should be mechanically drawn from the release and shipped in the same basic shape. It should not be a hand-written showcase that buyers will never encounter in the paid corpus.
The sample should include:
- one coherent unit rather than disconnected “best” files;
- the same IDs, manifest conventions, and checksum approach as the full product;
- exact sample statistics;
- quality and limitation notes; and
- a plain path from download to verification.
Gerra publishes a 479 KB Codebase Collection sample with one complete repository, 119 text files, 18,497 measured code lines, a cloneable 23-commit history, an anonymous commit index, five authentic patch pairs, and checksums. The path is deliberately plain: download the ZIP, verify SHA256SUMS, clone the bundle, then inspect the tree and every retained ref. The associated dataset page states which static checks passed and why runtime test success is not claimed.
9. Bind the release to exact bytes
The final package needs a version, immutable manifest, and complete checksum coverage. Re-run checks after extracting the actual delivery archive, not only against a staging directory.
Useful release evidence includes:
- SHA-256 for every member and the final archive;
- archive traversal, duplicate-entry, encryption, and symlink checks;
- Git bundle verification from the extracted copy;
- metric reconciliation against the extracted copy; and
- a dated independent review with clearly defined blocking classes.
The short buyer scorecard
A codebase dataset is ready for serious diligence when the supplier can show:
- coherent, named data units;
- reproducible counts;
- cloneable and reconciled history;
- measured duplication and generated content;
- tests described at the level actually executed;
- full-history privacy and secret controls;
- explicit provenance and permitted-use terms;
- a representative downloadable sample; and
- checksums and review evidence tied to the delivered archive.
Scale matters after these questions are answered. Before that, it is only a number.