LetTokenBurn
Sign in
Auto71%Harbor: pendingKnowledge & ReasoningSafety & Alignment

AI Consilium

AI Consilium: a Benchmark of LLM Clinical Multidisciplinary Second Opinions

A multidimensional benchmark using 300 NEJM-MedQA cases to evaluate LLM-generated clinical second opinions via physician-calibrated LLM judges across multiple criteria.

Paper

How to run

ltb · ai-consilium
$ npm i -g lettokenburn-cli
$ ltb bench info ai-consilium

External harness. No Harbor adapter has been generated for this benchmark yet. `ltb bench info` prints the known links.

Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs

Scores

8 sourced entries
HarnessSourceConfidenceLink
GPT-5.6 Sol2.8structuredIndependentmediumOct 15, 2026arxiv.org
GPT-oss 120B2.6structuredIndependentmediumOct 15, 2026arxiv.org
DeepSeek V4 Pro2.5structuredIndependentmediumOct 15, 2026arxiv.org
GLM-5.22.5structuredIndependentmediumOct 15, 2026arxiv.org
qwen3.5-27B2.5structuredIndependentmediumOct 15, 2026arxiv.org
Qwen3 235B2.4structuredIndependentmediumOct 15, 2026arxiv.org
google/gemma-4-31B-it2.4structuredIndependentmediumOct 15, 2026arxiv.org
Qwen3-Next-80B2.2structuredIndependentmediumOct 15, 2026arxiv.org

Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.

Reproduced by the community

All runs
Loading community runs…

About

The benchmark comprises 300 diagnostically resolved NEJM-MedQA cases, split into 110 for calibration and 190 for held-out evaluation. It assesses eight model pipelines generating role-specific opinions and interactive-consilium reports under free-form and structured conditions.

Scoring uses a physician-calibrated LLM Judge evaluating 3,040 final reports on criteria like factual grounding and safety. Structured reports generally scored higher, with GPT-5.6-Sol leading all components and achieving consensus fastest.