LetTokenBurn
Sign in
Auto86%Harbor: pendingMultimodalKnowledge & Reasoning

OccludeBench

OccludeBench: Can VLMs Find What's Missing?

A manually curated benchmark of 378 images testing VLMs' ability to recover fully occluded objects from direct physical cues (shadows/reflections) or broader scene context.

PaperRepo

How to run

ltb · occludebench
$ npm i -g lettokenburn-cli
$ ltb bench info occludebench

External harness. No Harbor adapter has been generated for this benchmark yet. `ltb bench info` prints the known links.

Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs

Scores

5 sourced entries
HarnessSourceConfidenceLink
Gemini 3.1 Pro78.3N/AIndependentmediumOct 9, 2026arxiv.org
Claude Opus 4.663.0N/AIndependentmediumOct 9, 2026arxiv.org
GPT-5.449.5N/AIndependentmediumOct 9, 2026arxiv.org
Qwen3-VL-8B-Instruct27.7N/AIndependentmediumOct 9, 2026arxiv.org
Llama 4 Maverick16.7N/AIndependentmediumOct 9, 2026arxiv.org

Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.

Reproduced by the community

All runs
Loading community runs…

About

OccludeBench contains 378 images split into direct cases (197) requiring reasoning from shadows or reflections, and inferential cases (181) relying on context. Objects are artificially occluded using bounding boxes or polygons.

Scoring uses an LLM-as-a-judge framework to evaluate open-ended responses against ground truth. VLMs average 42.7% accuracy on direct cases versus 62.7% on inferential ones, revealing a significant reasoning gap compared to humans.