This is not a Dataset
This is not a Dataset: A Large Negation Benchmark to Challenge Large Language Models
We introduce a large semi-automatically generated dataset of ~400,000 descriptive sentences about commonsense knowledge that can be true or false in which negation is present in about 2/3 of the corpus in different forms that we use to evaluate LLMs.
How to run
$ npm i -g lettokenburn-cli$ ltb bench pull this-is-not-a-dataset --n 50$ ltb bench run ./this-is-not-a-dataset -a claude-code -m anthropic/claude-opus-4-8$ ltb bench push
Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs
Scores
2 sourced entries| Harness | Source | Confidence | Link | |||
|---|---|---|---|---|---|---|
| Vicuna13B v1.1 | 95.7 | N/A | Independent | medium | Oct 24, 2023 | arxiv.org |
| Flan-T5-XXL | 94.1 | N/A | Independent | medium | Oct 24, 2023 | arxiv.org |
Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.
Reproduced by the community
All runsAbout
We introduce a large semi-automatically generated dataset of ~400,000 descriptive sentences about commonsense knowledge that can be true or false in which negation is present in about 2/3 of the corpus in different forms that we use to evaluate LLMs.
Source: ThisIsNotaDataset Image: ThisIsNotaDataset