LetTokenBurnChemArgus: Automated Scientific Diagnostics for Lab-Grade Multimodal Chemistry Reasoning
A diagnostic benchmark for expert-level multimodal chemistry reasoning that measures fine-grained stepwise derivation, visual understanding, and planning via three evaluation protocols.
$ npm i -g lettokenburn-cli$ ltb bench info chemargus
External harness. No Harbor adapter has been generated for this benchmark yet. `ltb bench info` prints the known links.
Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs
| Harness | Source | Confidence | Link | |||
|---|---|---|---|---|---|---|
| Claude Opus 5 | 61.5 | text only | Independent | medium | Oct 14, 2026 | arxiv.org |
| Gemini 3.8 Flash | 46.5 | text only | Independent | medium | Oct 14, 2026 | arxiv.org |
| Qwen3.8 Max | 36.0 | text only | Independent | medium | Oct 14, 2026 | arxiv.org |
| Kimi K3 | 34.5 | text only | Independent | medium | Oct 14, 2026 | arxiv.org |
| Grok-4.6 | 33.4 | text only | Independent | medium | Oct 14, 2026 | arxiv.org |
| GPT-5.6 Sol | 29.8 | text only | Independent | medium | Oct 14, 2026 | arxiv.org |
| MiniMax-M3 | 26.2 | text only | Independent | medium | Oct 14, 2026 | arxiv.org |
| DeepSeek V4 Pro | 25.0 | text only | Independent | medium | Oct 14, 2026 | arxiv.org |
| GLM-5.2 | 23.5 | text only | Independent | medium | Oct 14, 2026 | arxiv.org |
Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.
ChemArgus is a controlled dataset of high-difficulty chemistry problems featuring real images with controlled text descriptions. It targets expert-level open-ended answering and multimodal understanding.
Scoring uses rubric-grounded partial credit across subquestions. Three protocols (Full Problem, Sequential Carry, Oracle Scaffolded) isolate integration, propagation, and local reasoning errors to provide diagnostic attribution.