ARC-AGI-2
Harbor nativeARC-AGI-2: 更难的抽象推理
Verified100%Abstract Reasoning
- Samples
- 167
- Released
- 2025
- Scores
- 118
MS-Exam-Gen
MS-Exam-Gen: Source-Grounded Benchmark Construction for Evaluating LLMs on Textual Multiple Sclerosis MRI Knowledge
A source-grounded multiple-choice question benchmark with 3,058 items evaluating LLMs on textual Multiple Sclerosis MRI knowledge across 16 topics.
How to run
ltb · ms-exam-gen
$ npm i -g lettokenburn-cli$ ltb bench info ms-exam-gen
External harness. No Harbor adapter has been generated for this benchmark yet. `ltb bench info` prints the known links.
Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs
Scores
12 sourced entries| Harness | Source | Confidence | Link | |||
|---|---|---|---|---|---|---|
| Gemini 2.5 Flash | 89.7 | N/A | Independent | medium | Oct 5, 2026 | arxiv.org |
| Llama 4 Maverick | 88.5 | N/A | Independent | medium | Oct 5, 2026 | arxiv.org |
| DeepSeek Chat | 86.9 | N/A | Independent | medium | Oct 5, 2026 | arxiv.org |
| Claude Haiku 4.5 | 85.6 | N/A | Independent | medium | Oct 5, 2026 | arxiv.org |
| Gemini 2.5 Flash Lite | 84.4 | N/A | Independent | medium | Oct 5, 2026 | arxiv.org |
| Qwen3 14B | 83.3 | N/A | Independent | medium | Oct 5, 2026 | arxiv.org |
| GPT-5 nano (2025-08-07) | 81.2 | N/A | Independent | medium | Oct 5, 2026 | arxiv.org |
| GPT-4o-mini | 80.6 | N/A | Independent | medium | Oct 5, 2026 | arxiv.org |
| Qwen3 8B | 77.8 | N/A | Independent | medium | Oct 5, 2026 | arxiv.org |
| Llama-3.1-8B-Instruct | 73.1 | N/A | Independent | medium | Oct 5, 2026 | arxiv.org |
| Mistral | 71.8 | N/A | Independent | medium | Oct 5, 2026 | arxiv.org |
| Phi-4-mini-instruct | 46.9 | N/A | Independent | medium | Oct 5, 2026 | arxiv.org |
Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.
Reproduced by the community
All runsLoading community runs…
About
MS-Exam-Gen is a reproducible framework generating 3,058 MCQs from a 66-source corpus on MS-MRI criteria, protocols, and reporting. It employs expert-source indexing, automated quality audits, and same-family consistency screens to ensure auditability.
Scoring uses accuracy across 12 LLM endpoints, revealing a 42.8-point performance range (46.9% to 89.7%). The benchmark highlights position sensitivity and serves as a candidate set for item-level evaluation rather than a clinically certified exam.