Verified86%Harbor adapterMultimodalAbstract Reasoning

SpatialViz-Bench

Humans can imagine and manipulate visual images mentally, a capability known as \textit{spatial visualization}.

How to run

ltb · spatialviz-bench
$ npm i -g lettokenburn-cli
$ ltb bench pull spatialviz-bench --n 50
$ ltb bench run ./spatialviz-bench -a claude-code -m anthropic/claude-opus-4-8
$ ltb bench push

Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs

Scores

8 sourced entries
HarnessSourceConfidenceLink
Gemini 2.5 Pro44.7N/AIndependentmediumN/Aarxiv.org
OpenAI/o1-2024-12-17-high41.4N/AIndependentmediumN/Aarxiv.org
Gemini 2.5 Flash36.9N/AIndependentmediumN/Aarxiv.org
Llama-4-Scout-17B-16E-Instruct34.2N/AIndependentmediumN/Aarxiv.org
Claude 3.7 Sonnet Thinking (2025-02-19)33.9N/AIndependentmediumN/Aarxiv.org
Qwen2.5-VL-72B-Instruct33.3N/AIndependentmediumN/Aarxiv.org
Qwen-VL-Max32.0N/AIndependentmediumN/Aarxiv.org
Llama-4-Maverick-17B-128E-Instruct31.8N/AIndependentmediumN/Aarxiv.org

Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.

Reproduced by the community

All runs
Loading community runs…

About

Humans can imagine and manipulate visual images mentally, a capability known as \textit{spatial visualization}. While many multi-modal benchmarks assess reasoning on visible visual information, the ability to infer unseen relationships through spatial visualization remains insufficiently evaluated as a spatial skill. This reliance on publicly sourced problems from IQ tests or math competitions risks data contamination and compromises assessment reliability. To this end, we introduce \textbf{\textit{SpatialViz-Bench}}, a comprehensive multi-modal benchmark for \textit{spatial visualization} with \emph{12} tasks across \emph{4} sub-abilities, comprising \emph{1,180} programmatically generated problems, a scalable framework that allows for expansion to ensure fair and continuously reliable evaluations. Our evaluation of \emph{27} Multi-modal Large Language Models (MLLMs) reveals wide performance variations, demonstrates the benchmark's strong discriminative power, and uncovers counter-intuitive findings: Chain-of-Thought (CoT) prompting paradoxically degrades accuracy on open-source models. Through statistical and qualitative analysis of error types, SpatialViz-Bench demonstrates that state-of-the-art MLLMs exhibit deficiencies in \textit{spatial visualization} tasks, thereby addressing a significant lacuna in the field.