LetTokenBurnBenchmark of Linguistic Minimal Pairs for Polarity Items
A multilingual evaluation suite of over 12,000 NPI minimal pairs in 33 languages, testing grammaticality judgments for negative polarity items.
$ npm i -g lettokenburn-cli$ ltb bench info blimppi
External harness. No Harbor adapter has been generated for this benchmark yet. `ltb bench info` prints the known links.
Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs
| Harness | Source | Confidence | Link | |||
|---|---|---|---|---|---|---|
| Goldfish-70B | 97.2 | N/A | Independent | medium | Oct 9, 2026 | arxiv.org |
| Qwen3-27B | 96.2 | N/A | Independent | medium | Oct 9, 2026 | arxiv.org |
| Llama-3.1-70B-Instruct | 95.2 | N/A | Independent | medium | Oct 9, 2026 | arxiv.org |
| Apertus-32B | 94.8 | N/A | Independent | medium | Oct 9, 2026 | arxiv.org |
| Gemma 3 12B IT | 93.9 | N/A | Independent | medium | Oct 9, 2026 | arxiv.org |
| Gemma 3 4B | 93.4 | N/A | Independent | medium | Oct 9, 2026 | arxiv.org |
| Llama-3.1-8B-Instruct | 92.7 | N/A | Independent | medium | Oct 9, 2026 | arxiv.org |
| Goldfish-8B | 92.4 | N/A | Independent | medium | Oct 9, 2026 | arxiv.org |
| Aya-32B | 91.7 | N/A | Independent | medium | Oct 9, 2026 | arxiv.org |
| Apertus-14B | 91.0 | N/A | Independent | medium | Oct 9, 2026 | arxiv.org |
| Gemma 3 1B IT | 88.8 | N/A | Independent | medium | Oct 9, 2026 | arxiv.org |
| Apertus-8B | 87.0 | N/A | Independent | medium | Oct 9, 2026 | arxiv.org |
Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.
BLiMPPI contains naturalistic minimal pairs derived from Universal Dependencies treebanks across 33 languages. It targets the grammaticality of Negative Polarity Items (NPIs) in various licensing environments.
Models are evaluated on their ability to distinguish grammatical from ungrammatical sentences. Results indicate multilingual models struggle with lower-resource languages and non-n-gram solvable pairs.