ARC-AGI-2
Harbor nativeARC-AGI-2: 更难的抽象推理
Verified100%Abstract Reasoning
- Samples
- 167
- Released
- 2025
- Scores
- 120
VELD-QA
VELD-QA: A Large-Scale Dataset for Visual Evidence Grounding and Question Answering in Long Documents
VELD-QA is a large-scale benchmark with 15,000 human-authored QA pairs across 5,859 real-world PDFs, evaluating visual evidence grounding and question answering in long documents.
How to run
ltb · veld-qa
$ npm i -g lettokenburn-cli$ ltb bench info veld-qa
External harness. No Harbor adapter has been generated for this benchmark yet. `ltb bench info` prints the known links.
Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs
Scores
9 sourced entries| Harness | Source | Confidence | Link | |||
|---|---|---|---|---|---|---|
| Claude Opus 5.5 | 74.9 | top-3 rag (colpali) | Independent | medium | Oct 14, 2026 | arxiv.org |
| Claude Sonnet 5.5 | 74.9 | top-3 rag (colpali) | Independent | medium | Oct 14, 2026 | arxiv.org |
| GPT-6 Astra | 72.5 | top-3 rag (colpali) | Independent | medium | Oct 14, 2026 | arxiv.org |
| GPT-5.5 | 71.3 | top-3 rag (colqwen2) | Independent | medium | Oct 14, 2026 | arxiv.org |
| Muse Glimmer 30B | 66.4 | top-3 rag (colqwen2) | Independent | medium | Oct 14, 2026 | arxiv.org |
| Grok 4.20 | 60.8 | top-3 rag (colqwen2) | Independent | medium | Oct 14, 2026 | arxiv.org |
| Gemini 2.5 Pro | 59.4 | top-3 rag (colqwen2) | Independent | medium | Oct 14, 2026 | arxiv.org |
| Gemini 2.5 Flash | 51.7 | top-3 rag (colqwen2) | Independent | medium | Oct 14, 2026 | arxiv.org |
| Gemma 3 12B IT | 37.1 | top-3 rag (colqwen2) | Independent | medium | Oct 14, 2026 | arxiv.org |
Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.
Reproduced by the community
All runsLoading community runs…
About
The dataset contains 15,000 human-authored question-answer pairs across 5,859 real-world PDFs, covering chart, table, and complex-layout tasks with detailed visual evidence annotations. It enables separate evaluation of retrieval and interpretation capabilities in multimodal document understanding.
Evaluation of nine vision-language models reveals that evidence localization and visual interpretation are key bottlenecks, with oracle accuracy peaking at 81.7%. Enriching inputs with visual-element annotations significantly narrows the performance gap.