DROP
A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs
DROP is a QA dataset which tests comprehensive understanding of paragraphs.
How to run
$ npm i -g lettokenburn-cli$ ltb bench pull drop --n 50$ ltb bench run ./drop -a claude-code -m anthropic/claude-opus-4-8$ ltb bench push
Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs
Scores
0 sourced entriesNo sourced scores recorded yet.
Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.
Reproduced by the community
All runsAbout
DROP is a QA dataset which tests comprehensive understanding of paragraphs. In this crowdsourced, adversarially-created, 96k question-answering benchmark, a system must resolve multiple references in a question, map them onto a paragraph, and perform discrete operations over them (such as addition, counting, or sorting).
Available in EleutherAI lm-evaluation-harness as task family drop (1 tasks).