Waldo
Waldo: Query-language Preference under Cross-lingual Knowledge Disparities
A multilingual QA benchmark with 12K pairs from Wikipedia measuring query-language preference under knowledge gaps and conflicts across five languages.
How to run
$ npm i -g lettokenburn-cli$ ltb bench info waldo
External harness. No Harbor adapter has been generated for this benchmark yet. `ltb bench info` prints the known links.
Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs
Scores
3 sourced entries| Harness | Source | Confidence | Link | |||
|---|---|---|---|---|---|---|
| Qwen3 Max (2025-09-23) | 47.6 | N/A | Independent | medium | Sep 30, 2026 | arxiv.org |
| Aya-Expanse | 76.9 | N/A | Independent | medium | Sep 30, 2026 | arxiv.org |
| Llama 3.1 | 78.7 | N/A | Independent | medium | Sep 30, 2026 | arxiv.org |
Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.
Reproduced by the community
All runsAbout
Waldo constructs 12K QA pairs from Wikipedia revisions, targeting facts absent in English (gaps) or manually perturbed to create inconsistencies (conflicts). It evaluates if models favor query-language sources when evidence is heterogeneous.
Scoring quantifies query-language preference by comparing response distributions across languages. Findings show little preference in gaps, but strong alignment with query language in conflicts, mitigated by QLoRA training.