CavaBench
Cascaded Audio and Voice Assistant Benchmarking
A benchmark of 2,919 spoken financial queries across British accents to evaluate ASR robustness and downstream LLM tool-calling accuracy.
How to run
$ npm i -g lettokenburn-cli$ ltb bench info cavabench
External harness. No Harbor adapter has been generated for this benchmark yet. `ltb bench info` prints the known links.
Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs
Scores
8 sourced entries| Harness | Source | Confidence | Link | |||
|---|---|---|---|---|---|---|
| Distil-Whisper | 17.4 | N/A | Independent | medium | Oct 5, 2026 | arxiv.org |
| Whisper Turbo | 16.7 | N/A | Independent | medium | Oct 5, 2026 | arxiv.org |
| Parakeet Base Offline | 16.6 | N/A | Independent | medium | Oct 5, 2026 | arxiv.org |
| Parakeet FT Offline | 16.5 | N/A | Independent | medium | Oct 5, 2026 | arxiv.org |
| Nemotron | 16.1 | N/A | Independent | medium | Oct 5, 2026 | arxiv.org |
| Cohere Offline | 15.3 | N/A | Independent | medium | Oct 5, 2026 | arxiv.org |
| Qwen3-ASR-1.7B Chunking | 15.2 | N/A | Independent | medium | Oct 5, 2026 | arxiv.org |
| Qwen3-ASR-1.7B Offline | 11.1 | N/A | Independent | medium | Oct 5, 2026 | arxiv.org |
Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.
Reproduced by the community
All runsAbout
CavaBench contains 2,919 human-labeled audio utterances from 90 participants, covering four British accent groups and realistic financial tasks like budget tracking. It evaluates cascaded ASR-LLM pipelines under various acoustic conditions.
Scoring uses Word Error Rate (WER) for transcription and Exact Match (EM) for tool-calling. Findings show WER strongly predicts tool accuracy but fails to capture all task-level failures, highlighting accent-specific biases in financial voice assistants.