SiLTT
Polish–Silesian Translation Evaluation Dataset
A curated dataset of ~1.2k Polish-Silesian sentence pairs derived from literary texts to evaluate dialectal machine translation.
How to run
$ npm i -g lettokenburn-cli$ ltb bench info siltt
External harness. No Harbor adapter has been generated for this benchmark yet. `ltb bench info` prints the known links.
Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs
Scores
18 sourced entries| Harness | Source | Confidence | Link | |||
|---|---|---|---|---|---|---|
| Google Translate | 48.6 | N/A | Independent | medium | Oct 1, 2026 | arxiv.org |
| TransGemma-FT | 45.8 | N/A | Independent | medium | Oct 1, 2026 | arxiv.org |
| GPT-5.4 | 44.2 | N/A | Independent | medium | Oct 1, 2026 | arxiv.org |
| NLLB 54B MOE | 37.5 | N/A | Independent | medium | Oct 1, 2026 | arxiv.org |
| MADLAD-400-10B-MT | 37.2 | N/A | Independent | medium | Oct 1, 2026 | arxiv.org |
| Bielik-PL-11B-v3.0-IT | 35.1 | N/A | Independent | medium | Oct 1, 2026 | arxiv.org |
| NLLB 3.3B | 34.8 | N/A | Independent | medium | Oct 1, 2026 | arxiv.org |
| TransGemma | 32.1 | N/A | Independent | medium | Oct 1, 2026 | arxiv.org |
| Google Translate | 17.7 | N/A | Independent | medium | Oct 1, 2026 | arxiv.org |
| TransGemma-FT | 12.9 | N/A | Independent | medium | Oct 1, 2026 | arxiv.org |
| PLLuM-12B-nc-chat | 12.1 | N/A | Independent | medium | Oct 1, 2026 | arxiv.org |
| GPT-5.4 | 11.9 | N/A | Independent | medium | Oct 1, 2026 | arxiv.org |
| MADLAD-400-10B-MT | 9.8 | N/A | Independent | medium | Oct 1, 2026 | arxiv.org |
| NLLB 3.3B | 7.4 | N/A | Independent | medium | Oct 1, 2026 | arxiv.org |
| NLLB 54B MOE | 7.4 | N/A | Independent | medium | Oct 1, 2026 | arxiv.org |
| Bielik-PL-11B-v3.0-IT | 5.7 | N/A | Independent | medium | Oct 1, 2026 | arxiv.org |
| TransGemma | 5.1 | N/A | Independent | medium | Oct 1, 2026 | arxiv.org |
| PLLuM-12B-nc-chat | 0.1 | N/A | Independent | medium | Oct 1, 2026 | arxiv.org |
Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.
Reproduced by the community
All runsAbout
Content & Construction: SiLTT is a new evaluation benchmark for Polish-Silesian (szl) machine translation, addressing the scarcity and noise in existing resources like FLORES or BOUQuET. It consists of approximately 1,237 sentence pairs derived from human translations of literary works such as 'Cinderella' and 'Rapunzel', automatically aligned using LaBSE.
Scoring & Findings: The benchmark measures translation quality in low-resource dialectal settings where standard models fail due to orthographic variation. Results indicate that rule-based systems consistently outperform neural baselines on this specific dataset, highlighting the need for specialized approaches over large-scale noisy training.