ARC-AGI-2
Harbor nativeARC-AGI-2: 更难的抽象推理
Verified100%Abstract Reasoning
- Samples
- 167
- Released
- 2025
- Scores
- 118
Hob-VL
Agents Are Systems, Not Models: Rethinking Agentic Evaluation Benchmark
A benchmark evaluating coding agents on four scientific tasks (astrophysics/genomics) requiring the use of specialist models under varying configurations.
How to run
ltb · hob-vl
$ npm i -g lettokenburn-cli$ ltb bench info hob-vl
External harness. No Harbor adapter has been generated for this benchmark yet. `ltb bench info` prints the known links.
Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs
Scores
10 sourced entries| Harness | Source | Confidence | Link | |||
|---|---|---|---|---|---|---|
| Qwen3-VL-8B-Instruct | 50.6 | N/A | Independent | medium | Oct 1, 2026 | arxiv.org |
| llama-4 Scout | 50.0 | N/A | Independent | medium | Oct 1, 2026 | arxiv.org |
| DeepSeek Flash | 49.6 | N/A | Independent | medium | Oct 1, 2026 | arxiv.org |
| Claude Haiku 4.5 | 49.4 | N/A | Independent | medium | Oct 1, 2026 | arxiv.org |
| GPT-5.6 Luna | 49.0 | N/A | Independent | medium | Oct 1, 2026 | arxiv.org |
| Grok-4.3 | 48.8 | N/A | Independent | medium | Oct 1, 2026 | arxiv.org |
| Gemini 3.5 Flash Lite | 48.6 | N/A | Independent | medium | Oct 1, 2026 | arxiv.org |
| GLM-4.6V-FlashX | 48.6 | N/A | Independent | medium | Oct 1, 2026 | arxiv.org |
| Step 3.7 Flash | 14.6 | N/A | Independent | medium | Oct 1, 2026 | arxiv.org |
| Qwen-122B | 9.7 | N/A | Independent | medium | Oct 1, 2026 | arxiv.org |
Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.
Reproduced by the community
All runsLoading community runs…
About
Content & Construction: The benchmark consists of four scientific tasks from astrophysics and genomics. Agents operate in a sandbox with access to pre-built environments containing weights for various specialist models, which they must locate and execute via code generation rather than direct tool calls.
Scoring & Key Findings: Outcomes are scored against the published performance metrics of the original specialist model papers. The study finds that agent configuration (information, reasoning, verification, time budget, backbone) significantly impacts reliability, with information being the most influential factor.