SHAMS
SHAMI Annotated Multi-dialect Speech
A benchmark of 1,300 Levantine Arabic utterances across five varieties with four aligned tiers (audio, unvocalized text, diacritized text, phonetic transcription) for evaluating ASR, G2P, and A2P tasks.
How to run
$ npm i -g lettokenburn-cli$ ltb bench info shams
External harness. No Harbor adapter has been generated for this benchmark yet. `ltb bench info` prints the known links.
Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs
Scores
3 sourced entries| Harness | Source | Confidence | Link | |||
|---|---|---|---|---|---|---|
| Gemini-3.5-flash | 93.2 | N/A | Independent | medium | Oct 1, 2026 | arxiv.org |
| Whisper-Lev. | 35.6 | N/A | Independent | medium | Oct 1, 2026 | arxiv.org |
| GPT-4o | 18.0 | N/A | Independent | medium | Oct 1, 2026 | arxiv.org |
Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.
Reproduced by the community
All runsAbout
Content & Construction: SHAMS comprises 1,300 balanced utterances drawn from open audio corpora covering Urban/Rural Palestinian, Jordanian, Lebanese, and Syrian varieties. Each item features four aligned tiers: audio, unvocalized orthography, diacritized text, and phonetic transcription.
Scoring & Findings: The benchmark enables joint evaluation of tasks like automatic speech recognition (ASR), grapheme-to-phoneme conversion (G2P), and audio-to-phoneme. It provides baseline results across open and proprietary models stratified by variety to measure progress in Levantine Arabic technologies.