ARC-AGI-2
Harbor nativeARC-AGI-2: 更难的抽象推理
Verified100%Abstract Reasoning
- Samples
- 167
- Released
- 2025
- Scores
- 118
CASE longitudinal multimodal benchmark
CASE: Clinical Agents for Seeking Evidence Longitudinal Multimodal Benchmark
A longitudinal medical reasoning benchmark built on UK Biobank data comprising over 50k questions and multi-sequence MRI records from ~4.7k participants, requiring agents to autonomously gather evidence across visits.
How to run
ltb · case-longitudinal-multimodal-benchmark
$ npm i -g lettokenburn-cli$ ltb bench info case-longitudinal-multimodal-benchmark
External harness. No Harbor adapter has been generated for this benchmark yet. `ltb bench info` prints the known links.
Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs
Scores
13 sourced entries| Harness | Source | Confidence | Link | |||
|---|---|---|---|---|---|---|
| CASE | 75.1 | N/A | Independent | medium | Sep 30, 2026 | arxiv.org |
| Kimi K3 | 69.5 | N/A | Independent | medium | Sep 30, 2026 | arxiv.org |
| Qwen3.7 Plus | 67.7 | N/A | Independent | medium | Sep 30, 2026 | arxiv.org |
| Claude Opus 4.8 | 65.8 | N/A | Independent | medium | Sep 30, 2026 | arxiv.org |
| GPT-5.4-0305 | 65.7 | N/A | Independent | medium | Sep 30, 2026 | arxiv.org |
| Qwen3.8 Max | 58.3 | N/A | Independent | medium | Sep 30, 2026 | arxiv.org |
| Gemini-3.5-flash | 56.3 | N/A | Independent | medium | Sep 30, 2026 | arxiv.org |
| Lingshu-32B | 50.3 | N/A | Independent | medium | Sep 30, 2026 | arxiv.org |
| Qwen3-VL-8B-Instruct | 47.1 | N/A | Independent | medium | Sep 30, 2026 | arxiv.org |
| Lingshu-7B | 46.7 | N/A | Independent | medium | Sep 30, 2026 | arxiv.org |
| Kimi K2.6 | 42.0 | N/A | Independent | medium | Sep 30, 2026 | arxiv.org |
| MedGemma-1.5-4B | 41.4 | N/A | Independent | medium | Sep 30, 2026 | arxiv.org |
| MedGemma-27B | 27.5 | N/A | Independent | medium | Sep 30, 2026 | arxiv.org |
Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.
Reproduced by the community
All runsLoading community runs…
About
Content & Construction: The CASE benchmark consists of a patient-level tool-use harness where each question links to an environment containing clinical context (ICD-10 diagnoses) and heterogeneous multi-sequence MRIs from baseline and follow-up visits. Agents must autonomously select which organs, modalities, slices, and specialist tools to inspect before answering multiple-choice questions.
Scoring & Findings: Accuracy reflects the agent's ability to choose relevant evidence rather than just reasoning over fixed context. The benchmark supports agentic reinforcement learning; experiments show a Qwen3-VL-8B based agent achieves significant relative improvements in answer accuracy compared to frontier models like GPT-5.4 and Claude Opus 4.8.