LetTokenBurnJudge Under Plausible Explanation
A 250-query VQA-style benchmark fixing answers while varying explanations to measure how plausible text misleads multimodal judges in medical, chart, and hallucination domains.
$ npm i -g lettokenburn-cli$ ltb bench info jupe
External harness. No Harbor adapter has been generated for this benchmark yet. `ltb bench info` prints the known links.
Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs
| Harness | Source | Confidence | Link | |||
|---|---|---|---|---|---|---|
| Gemini 3 Flash Preview | 0.0 | N/A | Independent | medium | Oct 15, 2026 | arxiv.org |
| Gemini 3.1 Pro | 3.7 | N/A | Independent | medium | Oct 15, 2026 | arxiv.org |
| Gemini 3.1 Flash Lite | 7.7 | N/A | Independent | medium | Oct 15, 2026 | arxiv.org |
| GPT-5.4-mini | 11.1 | N/A | Independent | medium | Oct 15, 2026 | arxiv.org |
| GPT-5.4 | 14.8 | N/A | Independent | medium | Oct 15, 2026 | arxiv.org |
| InternVL3.5-8B-Instruct | 18.5 | N/A | Independent | medium | Oct 15, 2026 | arxiv.org |
| Qwen3-VL-8B-Instruct | 20.0 | N/A | Independent | medium | Oct 15, 2026 | arxiv.org |
| Qwen3-VL-32B | 22.2 | N/A | Independent | medium | Oct 15, 2026 | arxiv.org |
| GPT-5.6 Luna | 33.3 | N/A | Independent | medium | Oct 15, 2026 | arxiv.org |
| InternVL3.5-14B | 33.3 | N/A | Independent | medium | Oct 15, 2026 | arxiv.org |
| Gemma 3 4B | 52.0 | N/A | Independent | medium | Oct 15, 2026 | arxiv.org |
| Qwen3-VL-2B | 55.6 | N/A | Independent | medium | Oct 15, 2026 | arxiv.org |
Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.
JUPE comprises 250 visual-question-answering queries from medical, chart, and hallucination datasets. It isolates explanation effects by pairing fixed images, questions, and candidate answers with varying textual rationales.
Judges provide ordinal correctness labels. The benchmark measures 'plausibility capture,' finding that ordinary explanations significantly reduce the rejection rate of incorrect answers by shifting decision criteria rather than discriminability.