Aider Polyglot
Harbor nativeAider Polyglot: 6种语言diff编辑
- Samples
- 225
- Released
- 2024
- Scores
- 61
Humans can imagine and manipulate visual images mentally, a capability known as \textit{spatial visualization}.
$ npm i -g lettokenburn-cli$ ltb bench pull spatialviz-bench --n 50$ ltb bench run ./spatialviz-bench -a claude-code -m anthropic/claude-opus-4-8$ ltb bench push
Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs
| Harness | Source | Confidence | Link | |||
|---|---|---|---|---|---|---|
| Gemini 2.5 Pro | 44.7 | N/A | Independent | medium | N/A | arxiv.org |
| OpenAI/o1-2024-12-17-high | 41.4 | N/A | Independent | medium | N/A | arxiv.org |
| Gemini 2.5 Flash | 36.9 | N/A | Independent | medium | N/A | arxiv.org |
| Llama-4-Scout-17B-16E-Instruct | 34.2 | N/A | Independent | medium | N/A | arxiv.org |
| Claude 3.7 Sonnet Thinking (2025-02-19) | 33.9 | N/A | Independent | medium | N/A | arxiv.org |
| Qwen2.5-VL-72B-Instruct | 33.3 | N/A | Independent | medium | N/A | arxiv.org |
| Qwen-VL-Max | 32.0 | N/A | Independent | medium | N/A | arxiv.org |
| Llama-4-Maverick-17B-128E-Instruct | 31.8 | N/A | Independent | medium | N/A | arxiv.org |
Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.
Humans can imagine and manipulate visual images mentally, a capability known as \textit{spatial visualization}. While many multi-modal benchmarks assess reasoning on visible visual information, the ability to infer unseen relationships through spatial visualization remains insufficiently evaluated as a spatial skill. This reliance on publicly sourced problems from IQ tests or math competitions risks data contamination and compromises assessment reliability. To this end, we introduce \textbf{\textit{SpatialViz-Bench}}, a comprehensive multi-modal benchmark for \textit{spatial visualization} with \emph{12} tasks across \emph{4} sub-abilities, comprising \emph{1,180} programmatically generated problems, a scalable framework that allows for expansion to ensure fair and continuously reliable evaluations. Our evaluation of \emph{27} Multi-modal Large Language Models (MLLMs) reveals wide performance variations, demonstrates the benchmark's strong discriminative power, and uncovers counter-intuitive findings: Chain-of-Thought (CoT) prompting paradoxically degrades accuracy on open-source models. Through statistical and qualitative analysis of error types, SpatialViz-Bench demonstrates that state-of-the-art MLLMs exhibit deficiencies in \textit{spatial visualization} tasks, thereby addressing a significant lacuna in the field.