LetTokenBurnhttps://arxiv.org/abs/2406.06196 In this paper, we present the LingOly benchmark, a novel benchmark for advanced reasoning abilities in large language models.
$ npm i -g lettokenburn-cli$ ltb bench pull lingoly
External harness. pip install lm-eval && lm_eval --model <hf|vllm|openai-chat-completions> --model_args <model args> --tasks lingoly_context (lm-evaluation-harness task family 'lingoly') https://github.com/EleutherAI/lm-evaluation-harness
License is link-only: ltb bench pull fetches rows from the original host for local evaluation only and does not write a solution/ directory. Do not redistribute the generated tasks.
Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs
| Harness | Source | Confidence | Link | |||
|---|---|---|---|---|---|---|
| Claude Opus | 28.8 | N/A | Independent | medium | Jun 10, 2024 | arxiv.org |
| GPT-4o | 25.1 | N/A | Independent | medium | Jun 10, 2024 | arxiv.org |
| Gemini 1.5 Pro | 23.4 | N/A | Independent | medium | Jun 10, 2024 | arxiv.org |
| GPT-4 | 21.5 | N/A | Independent | medium | Jun 10, 2024 | arxiv.org |
| GPT-3.5 Turbo | 11.2 | N/A | Independent | medium | Jun 10, 2024 | arxiv.org |
| Mixtral 8x7B | 6.4 | N/A | Independent | medium | Jun 10, 2024 | arxiv.org |
| LLama-3 8B | 4.9 | N/A | Independent | medium | Jun 10, 2024 | arxiv.org |
| LLama-3 70B | 2.9 | N/A | Independent | medium | Jun 10, 2024 | arxiv.org |
| Gemma 7B | 2.2 | N/A | Independent | medium | Jun 10, 2024 | arxiv.org |
| Llama 2 70B | 1.1 | N/A | Independent | medium | Jun 10, 2024 | arxiv.org |
Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.
https://arxiv.org/abs/2406.06196 In this paper, we present the LingOly benchmark, a novel benchmark for advanced reasoning abilities in large language models. Using challenging Linguistic Olympiad puzzles, we evaluate (i) capabilities for in-context identification and generalisation of linguistic patterns in very low-resource or extinct languages, and (ii) abilities to follow complex task instructions. The LingOly benchmark covers more than 90 mostly low-resource languages, minimising issues of data contamination, and contains 1,133 problems across 6 formats and 5 levels of human difficulty. We assess performance with both direct accuracy and comparison to a no-context baseline to penalise memorisation. Scores from 11 state-of-the-art LLMs demonstrate the benchmark to be challenging, and models perform poorly on the higher difficulty problems. On harder problems, even the top model only achieved 38.7% accuracy, 24.7% improvement over the no-context baseline. Large closed models typically outperform open models, and in general, the higher resource the language, the better the scores. These results indicate, in absence of memorisation, true multi-step out-of-domain reasoning remains a challenge for current language models.
Available in EleutherAI lm-evaluation-harness as task family lingoly (2 tasks).