SEA-HELM Audio
SEA-HELM Audio: Evaluating Multilingual ASR-LLM Pipeline Performance Across SEA-HELM Subsets
A multilingual evaluation framework measuring performance degradation in cascaded ASR-LLM pipelines across five Southeast Asian languages using synthesized speech and rule-based verifiers.
How to run
$ npm i -g lettokenburn-cli$ ltb bench info sea-helm-audio
External harness. No Harbor adapter has been generated for this benchmark yet. `ltb bench info` prints the known links.
Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs
Scores
2 sourced entries| Harness | Source | Confidence | Link | |||
|---|---|---|---|---|---|---|
| Gemini-3.5-flash | 84.8 | sea-helm audio | Independent | medium | Oct 14, 2026 | arxiv.org |
| SEA-LION-4.5-27B | 80.6 | sea-helm audio | Independent | medium | Oct 14, 2026 | arxiv.org |
Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.
Reproduced by the community
All runsAbout
Extends the text-based SEA-HELM benchmark by synthesizing speech for prompts in Chinese, English, Malay, Indonesian, and Tamil. It evaluates open-weight and commercial ASR-LLM systems on multiple-choice and instruction-following tasks.
Scoring uses identical rule-based verifiers for text and speech inputs to quantify the 'speech gap'. Results show significant accuracy drops, particularly for Tamil, highlighting limitations of text-only benchmarks for voice-agent reliability.