LetTokenBurnSimuVerity: Benchmarking Agents for Engineering-Grade Simulink Model Generation
A benchmark of 101 text-to-executable Simulink model-generation tasks across ten engineering domains, evaluating agents on multidimensional engineering performance via native simulation.
$ npm i -g lettokenburn-cli$ ltb bench info simuverity
External harness. No Harbor adapter has been generated for this benchmark yet. `ltb bench info` prints the known links.
Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs
| Harness | Source | Confidence | Link | |||
|---|---|---|---|---|---|---|
| Claude Opus 4.8 | 42.9 | claude code | Independent | medium | Oct 1, 2026 | arxiv.org |
| GPT-5.5 | 41.7 | codex | Independent | medium | Oct 1, 2026 | arxiv.org |
| DeepSeek V4 Pro | 28.4 | claude code | Independent | medium | Oct 1, 2026 | arxiv.org |
| Qwen3.8 Max | 25.0 | claude code | Independent | medium | Oct 1, 2026 | arxiv.org |
| GLM-5.3 Flash | 5.0 | claude code | Independent | medium | Oct 1, 2026 | arxiv.org |
| Qwen3.8 27B-FP8 | 1.6 | claude code | Independent | medium | Oct 1, 2026 | arxiv.org |
Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.
SimuVerity comprises 101 tasks spanning ten engineering domains, each grounded by an executable-system profile and four families of native simulation scenarios. It evaluates artifact delivery, executability, and engineering qualification.
The hierarchical evaluator scores qualified models across six dimensions including accuracy and robustness. The best agent achieved only 42.86 overall, revealing significant gaps in satisfying multidimensional engineering requirements despite high structural similarity.