LetTokenBurnQuestion-answering NLI
The QNLI (Question-answering NLI) dataset is a Natural Language Inference dataset automatically derived from the Stanford Question Answering Dataset v1.1 (SQuAD).
$ npm i -g lettokenburn-cli$ ltb bench info qnli
External harness. No Harbor adapter has been generated for this benchmark yet. `ltb bench info` prints the known links.
Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs
| Harness | Source | Confidence | Link | |||
|---|---|---|---|---|---|---|
| T5-11B | 96.7 | N/A | Independent | medium | Oct 23, 2019 | arxiv.org |
| ERNIE 2.0 Large | 94.6 | N/A | Independent | medium | Jul 29, 2019 | arxiv.org |
| ERNIE 2.0 Base | 92.9 | N/A | Independent | medium | Jul 29, 2019 | arxiv.org |
| ERNIE | 91.3 | N/A | Independent | medium | May 17, 2019 | arxiv.org |
Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.
The QNLI (Question-answering NLI) dataset is a Natural Language Inference dataset automatically derived from the Stanford Question Answering Dataset v1.1 (SQuAD). SQuAD v1.1 consists of question-paragraph pairs, where one of the sentences in the paragraph (drawn from Wikipedia) contains the answer to the corresponding question (written by an annotator). The dataset was converted into sentence pair classification by forming a pair between each question and each sentence in the corresponding context, and filtering out pairs with low lexical overlap between the question and the context sentence. The task is to determine whether the context sentence contains the answer to the question. This modified version of the original task removes the requirement that the model select the exact answer, but also removes the simplifying assumptions that the answer is always present in the input and that lexical overlap is a reliable cue. The QNLI dataset is part of GLUE benchmark.