ARC-AGI-2
Harbor nativeARC-AGI-2: 更难的抽象推理
Verified100%Abstract Reasoning
- Samples
- 167
- Released
- 2025
- Scores
- 118
SQUARE-Bench
SQUARE-Bench: Evaluating Large Multimodal Models for AI-Generated Image Assessment
A comprehensive benchmark evaluating LMMs on four aspects (Semantics, Quality, Authenticity, Responsibility) using ~10K AIGIs and ~3K real images with expert QA pairs.
How to run
ltb · square-bench
$ npm i -g lettokenburn-cli$ ltb bench info square-bench
External harness. No Harbor adapter has been generated for this benchmark yet. `ltb bench info` prints the known links.
Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs
Scores
23 sourced entries| Harness | Source | Confidence | Link | |||
|---|---|---|---|---|---|---|
| Gemini 3 Pro | 91.0 | N/A | Independent | medium | Sep 29, 2026 | arxiv.org |
| Qwen3-VL-32B | 90.0 | N/A | Independent | medium | Sep 29, 2026 | arxiv.org |
| InternVL-3-5-38B | 88.6 | N/A | Independent | medium | Sep 29, 2026 | arxiv.org |
| MiniCPM-V-4-5 | 88.1 | N/A | Independent | medium | Sep 29, 2026 | arxiv.org |
| GLM-4.6V-Flash | 87.1 | N/A | Independent | medium | Sep 29, 2026 | arxiv.org |
| GPT-5.2-chat | 87.1 | N/A | Independent | medium | Sep 29, 2026 | arxiv.org |
| Qwen3-VL-8B-Instruct | 87.1 | N/A | Independent | medium | Sep 29, 2026 | arxiv.org |
| Claude Opus 4.5 (2025-11-01) | 86.7 | N/A | Independent | medium | Sep 29, 2026 | arxiv.org |
| Ovis2.5-9B | 86.7 | N/A | Independent | medium | Sep 29, 2026 | arxiv.org |
| InternVL-3.5-8B | 86.2 | N/A | Independent | medium | Sep 29, 2026 | arxiv.org |
| InternVL-3-8B | 85.9 | N/A | Independent | medium | Sep 29, 2026 | arxiv.org |
| InternVL-3-14B | 85.4 | N/A | Independent | medium | Sep 29, 2026 | arxiv.org |
| InternVL-3-5-4B | 85.0 | N/A | Independent | medium | Sep 29, 2026 | arxiv.org |
| InternVL-3-5-14B | 84.7 | N/A | Independent | medium | Sep 29, 2026 | arxiv.org |
| LLaVA-OneVision-1.5-8B | 84.0 | N/A | Independent | medium | Sep 29, 2026 | arxiv.org |
| Kimi-VL-A3B-Thinking | 80.6 | N/A | Independent | medium | Sep 29, 2026 | arxiv.org |
| Gemma 3 27B | 79.6 | N/A | Independent | medium | Sep 29, 2026 | arxiv.org |
| mPLUG-Owl3-7B | 75.2 | N/A | Independent | medium | Sep 29, 2026 | arxiv.org |
| Llama-3.2-11B-Vision-Instruct | 75.0 | N/A | Independent | medium | Sep 29, 2026 | arxiv.org |
| DeepSeek-VL-7B-Chat | 74.0 | N/A | Independent | medium | Sep 29, 2026 | arxiv.org |
| CogAgent-18B | 72.6 | N/A | Independent | medium | Sep 29, 2026 | arxiv.org |
| Llama3 LLaVA Next 8B | 72.1 | N/A | Independent | medium | Sep 29, 2026 | arxiv.org |
| DeepSeek-VL2-small | 56.1 | N/A | Independent | medium | Sep 29, 2026 | arxiv.org |
Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.
Reproduced by the community
All runsLoading community runs…
About
Content & Construction: SQUARE-Bench introduces a unified taxonomy of 38 sub-dimensions across Semantics, Quality, Authenticity, and Responsibility. It comprises nearly 10,000 curated AI-generated images from 22 diverse models plus over 3,000 real-world images, annotated with approximately 18,000 expert-driven question-answer pairs.
Scoring & Findings: The benchmark employs a dual-answer mechanism (Visual GT vs. Intended GT) to evaluate both LMM perception and T2I generation capabilities. Experiments on 23 Large Multimodal Models reveal that top proprietary models like Gemini-3-Pro outperform human experts, while open-source models occasionally surpass them in specific sub-dimensions.