PEDBench
PEDBench: Can Speech Models Detect Phonetic Errors?
A benchmark for phonetic error detection comprising 2,605 samples of natural child, acted, and synthetic speech with verified phoneme annotations.
How to run
$ npm i -g lettokenburn-cli$ ltb bench info pedbench
External harness. No Harbor adapter has been generated for this benchmark yet. `ltb bench info` prints the known links.
Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs
Scores
12 sourced entries| Harness | Source | Confidence | Link | |||
|---|---|---|---|---|---|---|
| Audio Flamingo 3 TIMIT+SC | 63.8 | N/A | Independent | medium | Oct 14, 2026 | arxiv.org |
| Qwen2.5-Omni-7B + TIMIT+SC | 63.5 | N/A | Independent | medium | Oct 14, 2026 | arxiv.org |
| BabAR | 59.5 | N/A | Independent | medium | Oct 14, 2026 | arxiv.org |
| Gemini 3.8 Flash | 58.3 | N/A | Independent | medium | Oct 14, 2026 | arxiv.org |
| Azure Speech | 50.0 | N/A | Independent | medium | Oct 14, 2026 | arxiv.org |
| HuPER | 48.8 | N/A | Independent | medium | Oct 14, 2026 | arxiv.org |
| Qwen2.5-Omni-7B | 46.5 | N/A | Independent | medium | Oct 14, 2026 | arxiv.org |
| W2V2P-XLSR53 | 44.1 | N/A | Independent | medium | Oct 14, 2026 | arxiv.org |
| PhoneticXeus | 43.4 | N/A | Independent | medium | Oct 14, 2026 | arxiv.org |
| GPT-Audio-1.5 | 41.7 | N/A | Independent | medium | Oct 14, 2026 | arxiv.org |
| ZIPA-CTC | 39.4 | N/A | Independent | medium | Oct 14, 2026 | arxiv.org |
| MultiIPA | 25.5 | N/A | Independent | medium | Oct 14, 2026 | arxiv.org |
Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.
Reproduced by the community
All runsAbout
PEDBench evaluates speech models on detecting phoneme-level substitutions, deletions, and insertions. It combines real-world child recordings, scripted actor productions, and synthetic TTS errors.
Scoring uses diagnosis-weighted precision, recall, and F1. Results show state-of-the-art models fail to reliably detect errors, exhibiting low precision and significant sim-to-real gaps.