BASIS-Bench
Bias And perSonality In Speech LLMs Benchmark
A benchmark evaluating speech LLMs on demographic robustness, stereotypical bias, and personalization across age and gender using spoken datasets.
How to run
$ npm i -g lettokenburn-cli$ ltb bench info basis-bench
External harness. No Harbor adapter has been generated for this benchmark yet. `ltb bench info` prints the known links.
Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs
Scores
8 sourced entries| Harness | Source | Confidence | Link | |||
|---|---|---|---|---|---|---|
| Qwen3-Omni | 93.4 | N/A | Independent | medium | Oct 9, 2026 | arxiv.org |
| Voxtral | 90.2 | N/A | Independent | medium | Oct 9, 2026 | arxiv.org |
| Step Audio2 | 89.7 | N/A | Independent | medium | Oct 9, 2026 | arxiv.org |
| MOSS-Audio | 89.4 | N/A | Independent | medium | Oct 9, 2026 | arxiv.org |
| AF-Next | 89.0 | N/A | Independent | medium | Oct 9, 2026 | arxiv.org |
| Qwen2.5-Omni | 87.1 | N/A | Independent | medium | Oct 9, 2026 | arxiv.org |
| AF-3 | 86.0 | N/A | Independent | medium | Oct 9, 2026 | arxiv.org |
| DeSTA 2.5 | 83.1 | N/A | Independent | medium | Oct 9, 2026 | arxiv.org |
Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.
Reproduced by the community
All runsAbout
BASIS-Bench evaluates speech LLMs using datasets like Spoken StereoSet+, ELIP-Pair, BASIS-Health, and BASIS-Product to measure demographic robustness, stereotypical bias, and personalization. It isolates acoustic, backbone, and interaction effects to diagnose sources of disparity.
Scoring involves measuring performance disparities and behavioral changes across demographic groups. Findings show models struggle with appropriate personalization from speech cues but exhibit amplified stereotypical behavior, particularly towards female and elderly speakers.