VeriCodeBench
Self-Spec Verifiable Code Generation Benchmark
A multilingual benchmark with 400 language-native problems in C, Java, Rust, and Python evaluating LLMs on self-spec verifiable code generation.
How to run
$ npm i -g lettokenburn-cli$ ltb bench info vericodebench
External harness. No Harbor adapter has been generated for this benchmark yet. `ltb bench info` prints the known links.
Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs
Scores
12 sourced entries| Harness | Source | Confidence | Link | |||
|---|---|---|---|---|---|---|
| Claude-Sonnet-5-VGCR | 100.0 | N/A | Independent | medium | Sep 30, 2026 | arxiv.org |
| Claude-Sonnet-5-CGS | 97.0 | N/A | Independent | medium | Sep 30, 2026 | arxiv.org |
| Claude-Sonnet-5-Direct | 90.0 | N/A | Independent | medium | Sep 30, 2026 | arxiv.org |
| Kimi-K2.7-Code-VGCR | 80.0 | N/A | Independent | medium | Sep 30, 2026 | arxiv.org |
| Qwen3.6-plus-VGCR | 79.0 | N/A | Independent | medium | Sep 30, 2026 | arxiv.org |
| Kimi-K2.7-Code-CGS | 61.0 | N/A | Independent | medium | Sep 30, 2026 | arxiv.org |
| Kimi-K2.7-Code-Direct | 56.0 | N/A | Independent | medium | Sep 30, 2026 | arxiv.org |
| Qwen3.6-plus-CGS | 54.0 | N/A | Independent | medium | Sep 30, 2026 | arxiv.org |
| Qwen3.6-plus-Direct | 54.0 | N/A | Independent | medium | Sep 30, 2026 | arxiv.org |
| DeepSeek V3.2+VGCR | 45.0 | N/A | Independent | medium | Sep 30, 2026 | arxiv.org |
| DeepSeek V3.2 | 17.0 | N/A | Independent | medium | Sep 30, 2026 | arxiv.org |
| DeepSeek V3.2+CGS | 17.0 | N/A | Independent | medium | Sep 30, 2026 | arxiv.org |
Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.
Reproduced by the community
All runsAbout
Content & Construction: The benchmark introduces 'self-spec' verifiable code generation where the LLM relies solely on its own generated specification throughout the process. It comprises 400 language-native problems across C, Java, Rust, and Python paired with native verification ecosystems.
Scoring & Findings: Evaluation measures specification coverage, code validity, and joint problem-level success. Results indicate that self-generated specifications remain a bottleneck; however, CodeNova substantially improves performance, enabling Claude Sonnet 5 to achieve the strongest results.