LetTokenBurn
Sign in
Auto86%Harbor: pendingCoding

QTestBench

QTestBench: A Test Framework for Evaluating LLM-Generated Quantum Code

A framework evaluating LLM-generated quantum test suites on 18 tasks (8 QHack, 10 Qiskit HumanEval) using oracle checkers to measure discrimination quality.

PaperRepo

How to run

ltb · qtestbench
$ npm i -g lettokenburn-cli
$ ltb bench info qtestbench

External harness. No Harbor adapter has been generated for this benchmark yet. `ltb bench info` prints the known links.

Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs

Scores

5 sourced entries
HarnessSourceConfidenceLink
Gemini 3 Pro1.0N/AIndependentmediumOct 9, 2026arxiv.org
Claude Opus 4.60.9N/AIndependentmediumOct 9, 2026arxiv.org
GPT-5.40.8N/AIndependentmediumOct 9, 2026arxiv.org
DeepSeek V3.20.4N/AIndependentmediumOct 9, 2026arxiv.org
Qwen3-Coder0.4N/AIndependentmediumOct 9, 2026arxiv.org

Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.

Reproduced by the community

All runs
Loading community runs…

About

QTestBench evaluates LLM-generated pytest suites for quantum code across PennyLane and Qiskit frameworks. It uses independent benchmark checkers as ground truth to score syntactic, semantic, and behavioral tests.

Scoring relies on executability, correctness acceptance, and bug-detection rejection metrics. Best models achieved a test quality of 0.71 on QHack and 0.34 on Qiskit HumanEval, showing executability overstates quality.