PolyCodeEval
PolyCodeEval: Benchmarking Multilingual Code Generation from Functions to Repositories
A unified multilingual benchmark with 2,590 code generation tasks across four granularities (function to repository) derived from 58 open-source repositories in five languages.
How to run
$ npm i -g lettokenburn-cli$ ltb bench info polycodeeval
External harness. No Harbor adapter has been generated for this benchmark yet. `ltb bench info` prints the known links.
Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs
Scores
14 sourced entries| Harness | Source | Confidence | Link | |||
|---|---|---|---|---|---|---|
| GPT | 71.7 | repocoder | Independent | medium | Oct 8, 2026 | arxiv.org |
| GPT | 68.9 | hcp | Independent | medium | Oct 8, 2026 | arxiv.org |
| DeepSeek Chat | 65.9 | repocoder | Independent | medium | Oct 8, 2026 | arxiv.org |
| Sonnet | 59.4 | aligncoder | Independent | medium | Oct 8, 2026 | arxiv.org |
| Sonnet | 58.7 | repocoder | Independent | medium | Oct 8, 2026 | arxiv.org |
| DeepSeek Chat | 56.0 | hcp | Independent | medium | Oct 8, 2026 | arxiv.org |
| DeepSeek Chat | 50.7 | llm direct | Independent | medium | Oct 8, 2026 | arxiv.org |
| GPT | 50.3 | aligncoder | Independent | medium | Oct 8, 2026 | arxiv.org |
| GPT | 48.8 | llm direct | Independent | medium | Oct 8, 2026 | arxiv.org |
| Sonnet | 43.2 | hcp | Independent | medium | Oct 8, 2026 | arxiv.org |
| Sonnet | 40.4 | llm direct | Independent | medium | Oct 8, 2026 | arxiv.org |
| DeepSeek Chat | 36.7 | aligncoder | Independent | medium | Oct 8, 2026 | arxiv.org |
| Claude Code | 22.4 | N/A | Independent | medium | Oct 8, 2026 | arxiv.org |
| Codex | 20.7 | N/A | Independent | medium | Oct 8, 2026 | arxiv.org |
Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.
Reproduced by the community
All runsAbout
PolyCodeEval comprises 2,590 tasks spanning functions (L3), files (L2), skeleton-conditioned repositories (L1), and full repositories (L0). It is constructed from 58 real, executable open-source repositories in Python, JavaScript, Java, Go, and C++.
Evaluation uses a unified execution-based protocol with integration procedures tailored to each granularity. Results indicate existing methods struggle with complete code fragments, achieving at most 71.7% correctness for functions and 31.0% for repositories.