OpenMTB-Audit
OpenMTB-Audit: Exposing Over-Refusal and Clinical Expert Perspectives in LLM-Based Molecular Tumor Board Safety Evaluation
An open-source benchmark of 500 synthetic NSCLC cases evaluating LLM safety calibration across four labels (Supported, Partially Supported, Unsupported, Insufficient Information).
How to run
$ npm i -g lettokenburn-cli$ ltb bench info openmtb-audit
External harness. No Harbor adapter has been generated for this benchmark yet. `ltb bench info` prints the known links.
Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs
Scores
2 sourced entries| Harness | Source | Confidence | Link | |||
|---|---|---|---|---|---|---|
| GPT-4o | 0.7 | prompt-checklist | Independent | medium | Oct 1, 2026 | arxiv.org |
| GPT-4o-mini | 0.7 | base llm | Independent | medium | Oct 1, 2026 | arxiv.org |
Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.
Reproduced by the community
All runsAbout
Comprises 500 synthetic non-small cell lung cancer cases (200 clean, 300 adversarial) derived from public evidence sources like CIViC and cBioPortal. Cases span five adversarial error categories to test clinical reasoning boundaries.
Evaluates label calibration and over-refusal rates rather than just correctness. Key findings show LLMs frequently collapse 'Partially Supported' cases, while the proposed MTB-AuditAgent reduces over-refusal to 6.7% with 91.2% accuracy.