medqa
Harbor: pendingAvailable in EleutherAI lm-evaluation-harness as task family medqa (2 tasks).
Verified86%Knowledge & Reasoning
- Samples
- 1,273
- Released
- 2020
- Scores
- 8
PubMedQA
A Dataset for Biomedical Research Question Answering
PubMedQA is a novel biomedical question answering (QA) dataset collected from PubMed abstracts.
How to run
ltb · pubmedqa
$ npm i -g lettokenburn-cli$ ltb bench pull pubmedqa
External harness. pip install lm-eval && lm_eval --model <hf|vllm|openai-chat-completions> --model_args <model args> --tasks pubmedqa (lm-evaluation-harness task family 'pubmedqa') https://github.com/EleutherAI/lm-evaluation-harness
Runs happen on your machine with your own subscription; only the score summary is pushed. CLI docs
Scores
12 sourced entries| Harness | Source | Confidence | Link | |||
|---|---|---|---|---|---|---|
| Med-PaLM 2 | 79.2 | N/A | Independent | medium | May 16, 2023 | arxiv.org |
| Flan-PaLM | 79.0 | N/A | Independent | medium | Dec 26, 2022 | arxiv.org |
| Codex 5-shot CoT | 78.2 | N/A | Independent | medium | Jul 17, 2022 | arxiv.org |
| Flan-PaLM | 77.2 | N/A | Independent | medium | Dec 26, 2022 | arxiv.org |
| Flan-T5-XXL | 76.8 | N/A | Independent | medium | May 17, 2024 | medrxiv.org |
| Claude 3 Opus | 75.8 | N/A | Independent | medium | Mar 4, 2024 | anthropic.com |
| Claude 3 Opus | 74.9 | N/A | Independent | medium | Mar 4, 2024 | anthropic.com |
| BLOOM | 73.6 | N/A | Independent | medium | Nov 16, 2022 | arxiv.org |
| Flan-PaLM | 67.6 | N/A | Independent | medium | Dec 26, 2022 | arxiv.org |
| PALM | 57.8 | N/A | Independent | medium | Dec 26, 2022 | arxiv.org |
| PALM | 55.0 | N/A | Independent | medium | Dec 26, 2022 | arxiv.org |
| PALM | 34.0 | N/A | Independent | medium | Dec 26, 2022 | arxiv.org |
Every number links to where it was published. Low-confidence rows are muted and need per-cell verification.
Reproduced by the community
All runsLoading community runs…
About
PubMedQA is a novel biomedical question answering (QA) dataset collected from PubMed abstracts. The task of PubMedQA is to answer research questions with yes/no/maybe (e.g.: Do preoperative statins reduce atrial fibrillation after coronary artery bypass grafting?) using the corresponding abstracts. PubMedQA has 1k expert-annotated, 61.2k unlabeled and 211.3k artificially generated QA instances. Each PubMedQA instance is composed of (1) a question which is either an existing research article title or derived from one, (2) a context which is the corresponding abstract without its conclusion, (3) a long answer, which is the conclusion of the abstract and, presumably, answers the research question, and (4) a yes/no/maybe answer which summarizes the conclusion.
Available in EleutherAI lm-evaluation-harness as task family pubmedqa (1 tasks).