LetTokenBurnChartBmkAgent: Harness-Governed Multi-Agent Construction of Chart QA Benchmarks from Sparse Error-Taxonomy Specifications
A dynamically constructed Chart QA benchmark with 300 samples targeting specific error taxonomies to evaluate MLLM perception and reasoning capabilities.
$ npm i -g lettokenburn-cli$ ltb bench info chartbmkagent
External harness. No Harbor adapter has been generated for this benchmark yet. `ltb bench info` prints the known links.
Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs
| Harness | Source | Confidence | Link | |||
|---|---|---|---|---|---|---|
| GPT-5.6 Sol | 84.3 | N/A | Independent | medium | Oct 6, 2026 | arxiv.org |
| GPT-5.5 | 82.7 | N/A | Independent | medium | Oct 6, 2026 | arxiv.org |
| Gemini 3.6 Flash | 80.7 | N/A | Independent | medium | Oct 6, 2026 | arxiv.org |
| Claude Opus 4.8 | 80.3 | N/A | Independent | medium | Oct 6, 2026 | arxiv.org |
| Gemini-3.5-flash | 80.3 | N/A | Independent | medium | Oct 6, 2026 | arxiv.org |
| Kimi K2.6 | 76.3 | N/A | Independent | medium | Oct 6, 2026 | arxiv.org |
| Nex-N2-Pro | 74.3 | N/A | Independent | medium | Oct 6, 2026 | arxiv.org |
| Qwen3.6 27B | 74.3 | N/A | Independent | medium | Oct 6, 2026 | arxiv.org |
| Grok 4.5 | 74.0 | N/A | Independent | medium | Oct 6, 2026 | arxiv.org |
| Grok-4.3 | 70.0 | N/A | Independent | medium | Oct 6, 2026 | arxiv.org |
| GLM-4.6V | 66.7 | N/A | Independent | medium | Oct 6, 2026 | arxiv.org |
| Kimi K2.5 | 63.0 | N/A | Independent | medium | Oct 6, 2026 | arxiv.org |
| Claude Opus 4.6 | 61.3 | N/A | Independent | medium | Oct 6, 2026 | arxiv.org |
| Claude Sonnet 5 | 50.7 | N/A | Independent | medium | Oct 6, 2026 | arxiv.org |
Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.
The benchmark consists of 300 Chart QA samples generated on-demand from sparse error-taxonomy specifications using a multi-agent framework. It targets coupled visual perception and numerical reasoning in charts.
Scoring uses accuracy across 17 MLLMs, ranging from 32.7% to 84.3%. Results demonstrate targeted diagnostic capability, with significant performance drops on error-specific samples compared to controls.