LetTokenBurn
Sign in
Auto71%Harbor: pendingSafety & AlignmentKnowledge & Reasoning

Sense and Sensitivity

Sense and Sensity: Benchmarking LLM Clinical Triage Recommendations with Physician Experts

A clinical triage benchmark evaluating the robustness of LLM recommendations against gender and tone perturbations using over 6,000 scenarios annotated by physicians.

Paper

How to run

ltb · sense-and-sensitivity
$ npm i -g lettokenburn-cli
$ ltb bench info sense-and-sensitivity

External harness. No Harbor adapter has been generated for this benchmark yet. `ltb bench info` prints the known links.

Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs

Scores

5 sourced entries
HarnessSourceConfidenceLink
GPT-4o91.4N/AIndependentmediumSep 29, 2026arxiv.org
Qwen Max (2025-01-25)86.5N/AIndependentmediumSep 29, 2026arxiv.org
DeepSeek Chat78.9N/AIndependentmediumSep 29, 2026arxiv.org
MedGemma75.9N/AIndependentmediumSep 29, 2026arxiv.org
Llama71.4N/AIndependentmediumSep 29, 2026arxiv.org

Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.

Reproduced by the community

All runs
Loading community runs…

About

Content & Construction: The benchmark consists of over 6,232 clinical scenarios derived from four datasets (AskDocs, OncQA, SCT, USMLE Derm). It includes baseline cases plus perturbed variants involving gender swaps/removals and tone changes. Annotations are provided by ten MD physicians for the baseline conditions.

Scoring & Findings: Models and physicians independently classify three binary triage decisions: self-management (Manage), in-person visit necessity (Visit), and resource allocation (Resource). Key findings indicate LLMs over-recommend care compared to physicians and exhibit higher sensitivity to clinically irrelevant text variations like gender or tone.