scrolls
Standardized CompaRison Over Long Language Sequences
Available in EleutherAI lm-evaluation-harness as task family scrolls (0 tasks).
How to run
ltb · scrolls
$ npm i -g lettokenburn-cli$ ltb bench pull scrolls
External harness. pip install lm-eval && lm_eval --model <hf|vllm|openai-chat-completions> --model_args <model args> --tasks scrolls (lm-evaluation-harness task family 'scrolls') https://github.com/EleutherAI/lm-evaluation-harness
Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs
Scores
1 sourced entryEvery number links to where it was published. Low-confidence rows are muted and need per-cell verification.
Reproduced by the community
All runsLoading community runs…
About
Available in EleutherAI lm-evaluation-harness as task family scrolls (0 tasks).