LongBench
In this paper, we introduce LongBench, the first bilingual, multi-task benchmark for long context understanding, enabling a more rigorous evaluation of long context understanding.
How to run
$ npm i -g lettokenburn-cli$ ltb bench pull longbench --n 50$ ltb bench run ./longbench -a claude-code -m anthropic/claude-opus-4-8$ ltb bench push
Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs
Scores
0 sourced entriesNo sourced scores recorded yet.
Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.
Reproduced by the community
All runsAbout
In this paper, we introduce LongBench, the first bilingual, multi-task benchmark for long context understanding, enabling a more rigorous evaluation of long context understanding. LongBench comprises 21 datasets across 6 task categories in both English and Chinese, with an average length of 6,711 words (English) and 13,386 characters (Chinese). These tasks cover key long-text application areas including single-doc QA, multi-doc QA, summarization, few-shot learning, synthetic tasks, and code completion. All datasets in LongBench are standardized into a unified format, allowing for effortless automatic evaluation of LLMs
Available in EleutherAI lm-evaluation-harness as task family longbench (34 tasks).