VisAudit
VisAudit: Evaluating Multimodal Agents for Visual Diagnosis and Repair
A benchmark with 2,200 instances (1,900 flawed, 300 correct) evaluating multimodal agents on autonomous visualization diagnosis, repair, and verification across 21 chart types.
How to run
$ npm i -g lettokenburn-cli$ ltb bench info visaudit
External harness. No Harbor adapter has been generated for this benchmark yet. `ltb bench info` prints the known links.
Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs
Scores
7 sourced entries| Harness | Source | Confidence | Link | |||
|---|---|---|---|---|---|---|
| Gemini | 68.0 | N/A | Independent | medium | Oct 1, 2026 | arxiv.org |
| Claude | 63.1 | N/A | Independent | medium | Oct 1, 2026 | arxiv.org |
| DeepSeek Chat | 63.1 | N/A | Independent | medium | Oct 1, 2026 | arxiv.org |
| Qwen Max (2025-01-25) | 60.2 | N/A | Independent | medium | Oct 1, 2026 | arxiv.org |
| Gemma | 57.1 | N/A | Independent | medium | Oct 1, 2026 | arxiv.org |
| Mistral | 56.7 | N/A | Independent | medium | Oct 1, 2026 | arxiv.org |
| Ministral | 35.1 | N/A | Independent | medium | Oct 1, 2026 | arxiv.org |
Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.
Reproduced by the community
All runsAbout
VisAudit contains 1,900 flawed and 300 correct chart instances constructed via controlled perturbations of validated source visualizations. It spans 21 chart types and 10 flaw categories, providing auxiliary evidence like source data and code.
The benchmark uses three tracks: diagnosed repair, autonomous repair, and open-world verification. Evaluation measures executability, recovery, preservation, and correct-chart recognition, revealing that top models recover only 47.4% of flaws autonomously.