Visual CoT
Harbor: pendingMulti-Modal Large Language Models (MLLMs) have demonstrated impressive performance in various VQA tasks.
- Samples
- N/A
- Released
- 2024
- Scores
- 0
Visual Sanad
Visual Sanad: Auditing Vision-Language Models for False Acceptance of Qur'anic Citations
A 2,400-image benchmark testing if VLMs falsely accept incorrect Qur'anic verse attributions under varied visual framings.
How to run
ltb · visual-sanad
$ npm i -g lettokenburn-cli$ ltb bench info visual-sanad
External harness. No Harbor adapter has been generated for this benchmark yet. `ltb bench info` prints the known links.
Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs
Scores
12 sourced entries| Harness | Source | Confidence | Link | |||
|---|---|---|---|---|---|---|
| GPT-5.5 | 99.8 | N/A | Independent | medium | Oct 15, 2026 | arxiv.org |
| GPT-5.6-terra | 95.1 | N/A | Independent | medium | Oct 15, 2026 | arxiv.org |
| GPT-4.1 | 83.7 | N/A | Independent | medium | Oct 15, 2026 | arxiv.org |
| GPT-4o | 83.4 | N/A | Independent | medium | Oct 15, 2026 | arxiv.org |
| Qwen3-VL-32B | 12.2 | N/A | Independent | medium | Oct 15, 2026 | arxiv.org |
| InternVL3.5-38B | 7.6 | N/A | Independent | medium | Oct 15, 2026 | arxiv.org |
| Qwen2.5-VL-32B | 5.1 | N/A | Independent | medium | Oct 15, 2026 | arxiv.org |
| Qwen3-VL-8B-Instruct | 3.2 | N/A | Independent | medium | Oct 15, 2026 | arxiv.org |
| Qwen2.5-VL-7B | 0.9 | N/A | Independent | medium | Oct 15, 2026 | arxiv.org |
| InternVL3.5-8B-Instruct | 0.7 | N/A | Independent | medium | Oct 15, 2026 | arxiv.org |
| Gemma-4-12B-it | 0.5 | N/A | Independent | medium | Oct 15, 2026 | arxiv.org |
| MiniCPM-V-4-5 | 0.5 | N/A | Independent | medium | Oct 15, 2026 | arxiv.org |
Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.
Reproduced by the community
All runsLoading community runs…
About
The dataset contains 2,400 rendered images derived from 100 English Qur'an verses, featuring four factual states and six visual conditions. It evaluates whether models verify quote-reference pairs correctly despite decorative or authority-framed layouts.
Scoring focuses on false-accept rate on invalid trials and discrimination. Results show proprietary models vary in performance, while many open-weight models default to 'VALID', revealing weak citation discrimination and susceptibility to visual cues.