CARES
CARES: A Controlled Synthetic Benchmark of Speaker Reactions to Sound
A comprehensive evaluation of trustworthiness in medical large vision language models
How to run
$ npm i -g lettokenburn-cli$ ltb bench info cares
External harness. No Harbor adapter has been generated for this benchmark yet. `ltb bench info` prints the known links.
Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs
Scores
7 sourced entries| Harness | Source | Confidence | Link | |||
|---|---|---|---|---|---|---|
| Qwen3-Omni | 39.0 | N/A | Independent | medium | Oct 7, 2026 | arxiv.org |
| Whisper + LLM | 36.5 | N/A | Independent | medium | Oct 7, 2026 | arxiv.org |
| MiMo-Audio | 32.8 | N/A | Independent | medium | Oct 7, 2026 | arxiv.org |
| MOSS-Audio | 31.4 | N/A | Independent | medium | Oct 7, 2026 | arxiv.org |
| Audio Flamingo Next | 29.7 | N/A | Independent | medium | Oct 7, 2026 | arxiv.org |
| Kimi-Audio | 25.6 | N/A | Independent | medium | Oct 7, 2026 | arxiv.org |
| MiDashengLM | 24.1 | N/A | Independent | medium | Oct 7, 2026 | arxiv.org |
Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.
Reproduced by the community
All runsAbout
Artificial intelligence has significantly impacted medical applications, particularly with the advent of Medical Large Vision Language Models (Med-LVLMs), sparking optimism for the future of automated and personalized healthcare. However, the trustworthiness of Med-LVLMs remains unverified, posing significant risks for future model deployment. In this paper, we introduce CARES and aim to comprehensively evaluate the Trustworthiness of Med-LVLMs across the medical domain. We assess the trustworthiness of Med-LVLMs across five dimensions, including trustfulness, fairness, safety, privacy, and robustness. CARES comprises about 41K question-answer pairs in both closed and open-ended formats, covering 16 medical image modalities and 27 anatomical regions. Our analysis reveals that the models consistently exhibit concerns regarding trustworthiness, often displaying factual inaccuracies and failing to maintain fairness across different demographic groups. Furthermore, they are vulnerable to attacks and demonstrate a lack of privacy awareness. We publicly release our benchmark and code in https://github.com/richard-peng-xia/CARES.