BLiMPPI
Harbor: pendingA multilingual evaluation suite of over 12,000 NPI minimal pairs in 33 languages, testing grammaticality judgments for negative polarity items.
- Samples
- 12,000
- Released
- 2026
- Scores
- 12
OTel
Open Telco (OTel)
A telecom AI benchmark evaluating retrieval, reranking, and LLM correctness using held-out partitions from derived datasets.
How to run
ltb · otel
$ npm i -g lettokenburn-cli$ ltb bench info otel
External harness. No Harbor adapter has been generated for this benchmark yet. `ltb bench info` prints the known links.
Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs
Scores
13 sourced entries| Harness | Source | Confidence | Link | |||
|---|---|---|---|---|---|---|
| Gemma 3 12B IT | 78.3 | N/A | Independent | medium | Oct 6, 2026 | arxiv.org |
| RNJ-1-Instruct | 72.5 | N/A | Independent | medium | Oct 6, 2026 | arxiv.org |
| LFM2.5-1.2B-Instruct | 66.4 | N/A | Independent | medium | Oct 6, 2026 | arxiv.org |
| Gemma 3 4B | 66.2 | N/A | Independent | medium | Oct 6, 2026 | arxiv.org |
| gpt-oss-20b | 61.4 | N/A | Independent | medium | Oct 6, 2026 | arxiv.org |
| Qwen3 14B | 60.7 | N/A | Independent | medium | Oct 6, 2026 | arxiv.org |
| Qwen3 8B | 60.4 | N/A | Independent | medium | Oct 6, 2026 | arxiv.org |
| OLMo-3-7B | 57.4 | N/A | Independent | medium | Oct 6, 2026 | arxiv.org |
| Mistral-3-3B | 56.9 | N/A | Independent | medium | Oct 6, 2026 | arxiv.org |
| Qwen3 1.7B | 52.8 | N/A | Independent | medium | Oct 6, 2026 | arxiv.org |
| Qwen-3-0.6B | 49.0 | N/A | Independent | medium | Oct 6, 2026 | arxiv.org |
| Gemma 3 1B IT | 48.3 | N/A | Independent | medium | Oct 6, 2026 | arxiv.org |
| Gemma 3 270M IT | 22.2 | N/A | Independent | medium | Oct 6, 2026 | arxiv.org |
Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.
Reproduced by the community
All runsLoading community runs…
About
OTel provides derived datasets for retrieval, reranking, instruction tuning, and safety, built from 1.1M raw telecom documents. It includes 30 post-trained baselines evaluated on held-out partitions.
Scoring uses NDCG@10 for embeddings, MRR@10 for rerankers, and LLM-as-judge correctness. Baselines achieve 93.1% NDCG@10, 0.947 MRR@10, and 87.8% correctness.