SolBehave
SolBehave: Evaluating the Behavioural Correctness of LLM-Generated Smart Contracts
A benchmark of 250 real Ethereum contracts evaluating LLM-generated code via differential fuzzing, historical replay, and DCR-guided synthetic replay.
How to run
$ npm i -g lettokenburn-cli$ ltb bench info solbehave
External harness. No Harbor adapter has been generated for this benchmark yet. `ltb bench info` prints the known links.
Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs
Scores
7 sourced entries| Harness | Source | Confidence | Link | |||
|---|---|---|---|---|---|---|
| google/gemma-4-31B-it | 92.7 | N/A | Independent | medium | Oct 13, 2026 | arxiv.org |
| Qwen3.6 27B | 71.3 | N/A | Independent | medium | Oct 13, 2026 | arxiv.org |
| Qwen3-Coder-30B | 61.8 | N/A | Independent | medium | Oct 13, 2026 | arxiv.org |
| Gemma-4-12B-it | 57.9 | N/A | Independent | medium | Oct 13, 2026 | arxiv.org |
| Ornith-35B | 52.1 | N/A | Independent | medium | Oct 13, 2026 | arxiv.org |
| Codestral-22B | 39.9 | N/A | Independent | medium | Oct 13, 2026 | arxiv.org |
| Devstral-24B | 26.4 | N/A | Independent | medium | Oct 13, 2026 | arxiv.org |
Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.
Reproduced by the community
All runsAbout
SolBehave comprises 250 reproducible Ethereum smart contracts with deployment metadata and extracted Dynamic Condition Response graphs. It evaluates behavioral fidelity using three oracles: differential fuzzing, historical transaction replay, and DCR-guided synthetic transaction replay.
Scoring measures divergence from original contract behavior rather than just compilation success. Results show compilation is a poor proxy for correctness, with significant behavioral divergence detected across most models, highlighting the necessity of behavioral testing.