LetTokenBurn
Sign in
Auto57%Harbor: pendingKnowledge & Reasoning

StoryCloze

https://arxiv.org/abs/1604.01696 Homepage: https://cs.rochester.edu/nlp/rocstories/ Available in EleutherAI lm-evaluation-harness as task family storycloze (2 tasks).

PaperDatasetLeaderboardHomepage

How to run

ltb · storycloze
$ npm i -g lettokenburn-cli
$ ltb bench pull storycloze

External harness. pip install lm-eval && lm_eval --model <hf|vllm|openai-chat-completions> --model_args <model args> --tasks storycloze_2016 (lm-evaluation-harness task family 'storycloze') https://github.com/EleutherAI/lm-evaluation-harness

License is link-only: ltb bench pull fetches rows from the original host for local evaluation only and does not write a solution/ directory. Do not redistribute the generated tasks.

Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs

Scores

4 sourced entries
HarnessSourceConfidenceLink
GPT-3-175B87.7N/AIndependentmediumMay 28, 2020arxiv.org
PaLM 2-L87.4N/AIndependentmediumMay 17, 2023arxiv.org
PaLM 2-M86.7N/AIndependentmediumMay 17, 2023arxiv.org
PaLM 2-S85.6N/AIndependentmediumMay 17, 2023arxiv.org

Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.

Reproduced by the community

All runs
Loading community runs…

About

https://arxiv.org/abs/1604.01696 Homepage: https://cs.rochester.edu/nlp/rocstories/

Available in EleutherAI lm-evaluation-harness as task family storycloze (2 tasks).