LetTokenBurnVisual Signal vs. Language Prior Benchmark
A multimodal benchmark measuring language prior interference in MLLMs using rare-but-real scenarios across six domains with model-aware metrics.
$ npm i -g lettokenburn-cli$ ltb bench info visip-bench
External harness. No Harbor adapter has been generated for this benchmark yet. `ltb bench info` prints the known links.
Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs
| Harness | Source | Confidence | Link | |||
|---|---|---|---|---|---|---|
| InternVL3.5-38B | 30.0 | N/A | Independent | medium | Oct 15, 2026 | arxiv.org |
| InternVL3-38B | 36.8 | N/A | Independent | medium | Oct 15, 2026 | arxiv.org |
| InternVL3-78B | 37.0 | N/A | Independent | medium | Oct 15, 2026 | arxiv.org |
| Qwen2.5-VL-72B-Instruct | 37.0 | N/A | Independent | medium | Oct 15, 2026 | arxiv.org |
| InternVL3.5-14B | 38.6 | N/A | Independent | medium | Oct 15, 2026 | arxiv.org |
| Qwen2.5-VL-32B | 42.8 | N/A | Independent | medium | Oct 15, 2026 | arxiv.org |
| InternVL3.5-8B-Instruct | 47.2 | N/A | Independent | medium | Oct 15, 2026 | arxiv.org |
| InternVL3-14B | 48.4 | N/A | Independent | medium | Oct 15, 2026 | arxiv.org |
| InternVL3.5-2B | 51.5 | N/A | Independent | medium | Oct 15, 2026 | arxiv.org |
| Qwen2.5-VL-7B | 55.4 | N/A | Independent | medium | Oct 15, 2026 | arxiv.org |
| Qwen3-VL-2B | 59.4 | N/A | Independent | medium | Oct 15, 2026 | arxiv.org |
| InternVL3-2B | 62.8 | N/A | Independent | medium | Oct 15, 2026 | arxiv.org |
| Qwen2.5-VL-3B | 63.2 | N/A | Independent | medium | Oct 15, 2026 | arxiv.org |
Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.
ViSiP-Bench evaluates MLLM visual perception against language priors using 1,200 rare-but-real scenarios across six categories (e.g., Physical Properties, Scientific). Images are synthesized or retrieved to avoid biases.
It uses dynamic prior induction and filters chance successes. Key metrics include Prior Adherence Rate (PAR) and Visual Information Gain (VIG), revealing pervasive prior adherence that does not improve with model scale.