VHDL-REPOBENCH
VHDL-REPOBENCH: A Repository-Level Benchmark for Evaluating Large Language Models on VHDL Design Generation
A repository-level benchmark with ~2.5k VHDL files and ~500 testbenches evaluating LLMs on syntax, semantics, and functional correctness via simulation.
How to run
$ npm i -g lettokenburn-cli$ ltb bench info vhdl-repobench
External harness. No Harbor adapter has been generated for this benchmark yet. `ltb bench info` prints the known links.
Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs
Scores
5 sourced entries| Harness | Source | Confidence | Link | |||
|---|---|---|---|---|---|---|
| GPT-4o | 60.9 | N/A | Independent | medium | Oct 4, 2026 | arxiv.org |
| Qwen2.5-72B-Instruct | 53.1 | N/A | Independent | medium | Oct 4, 2026 | arxiv.org |
| LLama-3 70B | 48.9 | N/A | Independent | medium | Oct 4, 2026 | arxiv.org |
| CodeLlama-70B | 40.3 | N/A | Independent | medium | Oct 4, 2026 | arxiv.org |
| Verigen | 24.8 | N/A | Independent | medium | Oct 4, 2026 | arxiv.org |
Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.
Reproduced by the community
All runsAbout
Curated from ~100 open-source GitHub repositories, preserving full project hierarchies, cross-file dependencies, and verification artifacts for realistic VHDL design tasks.
Evaluation uses GHDL compilation and testbench simulation to measure functional correctness across line prediction, block completion, module generation, and summarization, revealing challenges in multi-file reasoning.