LetTokenBurn
Sign in
Auto86%Harbor: pendingCodingAgentic / Tool Use

Zero2Repo

Zero2Repo: Can Coding Agents Build Repositories from Scratch?

A benchmark evaluating coding agents' ability to build complete software repositories in native ecosystems (Python, TypeScript, Go, C++) from scratch using sealed acceptance tests.

PaperRepoHomepage

How to run

ltb · zero2repo
$ npm i -g lettokenburn-cli
$ ltb bench info zero2repo

External harness. No Harbor adapter has been generated for this benchmark yet. `ltb bench info` prints the known links.

Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs

Scores

3 sourced entries
HarnessSourceConfidenceLink
GPT-6 Astra90.9codexIndependentmediumSep 29, 2026arxiv.org
Claude Opus 5.581.8claude codeIndependentmediumSep 29, 2026arxiv.org
Grok-4.763.6cursor cliIndependentmediumSep 29, 2026arxiv.org

Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.

Reproduced by the community

All runs
Loading community runs…

About

Content & Construction: Zero2Repo evaluates agents on building full repositories starting from an empty workspace. Tasks are derived from real open-source projects via a language-agnostic pipeline that neutralizes names and hides implementation details. Agents receive product requirements documents (Phys. Rev. D) and interface contracts but must handle dependencies, build configurations, and packaging themselves without seeing acceptance tests until submission.

Scoring & Key Findings: Evaluation uses isolated judge containers to run hidden acceptance tests only after explicit submission, assigning a binary reward if all pass with no LLM judges involved. On 11 tasks from well-known projects, the strongest agents solved only 10; failures often trace to single omissions or low-frequency rules rather than missing subsystems.