Overview
| Name | PubMedQA · aliases: PubMedQA, PQA-L, PQA-U, PQA-A |
|---|---|
| Full name | The name has no expanded form. |
| Ambiguous names (not auto-linked) | None |
| Family | Not a family entry. Single static dataset; no family members. |
| Domain | Text, LLMs & agents |
| Inclusion | Original 30 |
| Type | B E · Static text dataset, a routine LLM eval row |
| Evidence level | L1 |
| Publisher | University of Pittsburgh, Carnegie Mellon University (Academic lab) |
| First release | · paper v1 Basis: arXiv v1 of the PubMedQA paper (2019-09-13); GitHub initial commit 2019-08-23 kept as a note only (repo creation excluded as evidence). source |
| Other milestones |
|
| Latest update | · maintenance only Basis: README commit on the data repository README edit on the data repository (2023-04-18); dataset frozen since 2019. Most recent sourced event, not a claim that nothing happened later. source The most recent sourced event, not a claim that nothing changed since. |
| Next expected | Dataset frozen; nothing announced by the authors. |
| Status | ○ Frozen · Saturated; newest official-board entry dated 2024-04-18 |
| Scale | PQA-L 1.0k expert-labeled (500 test), PQA-U 61.2k, PQA-A 211.3k.
|
| Wet-lab refresh | Static data · None; fixed expert-labelled text data. |
| Own wet lab needed to evaluate a new model | No, run it as a download |
| Models reported | 2 counted (self 2, unconfirmed 6) |
| Unverified fields | † release_milestones.first_public_evidence, ranking.entries[5], models_reported[0] |
| As of | 2026-11-16 |
What it measures
- Task: Answering yes/no/maybe questions from PubMed abstracts, testing reading comprehension of study conclusions.
- Truth: Two annotators' classification of the authors' conclusion (PQA-L); authors estimate about 1% residual error.
- Primary metric: Accuracy (reasoning-required setting)
- Secondary metrics: Macro-F1
- Version traps: State setting (reasoning-required or free), zero/few-shot, fine-tuning, retrieval or ensembling, and whether the train split was used.
Current ranking
Board
| Metric | Accuracy (%) (macro-F1 mostly not reported), reasoning-required setting |
|---|---|
| Eval set | PQA-L test (500 questions) |
| Snapshot | stale |
| Board | https://pubmedqa.github.io |
| Retrieved | 2026-11-16 |
Caveat. Newest board entry 2024-04-18; no later frontier reasoning models. Mixes zero-shot, few-shot, fine-tuned and ensembles, all self-submitted; top five within ~1.4 points vs ~±1.9-point binomial SE on 500 items.
Shown in the overview table: Board
| Rank | Model | Score (verbatim) | Metric · eval set / protocol | Source type · date | Source and note |
|---|---|---|---|---|---|
| 1 | GPT-4 (Medprompt) | 82.0 | Accuracy (%) (macro-F1 mostly not reported), reasoning-required setting PQA-L test (500 questions) | Board | source Overall top-1; prompting plus ensembling, self-submitted. |
| 6 | GPT-4-base | 80.4 | Accuracy (%) (macro-F1 mostly not reported), reasoning-required setting PQA-L test (500 questions) | Board | source |
| 7 | Claude 3 | 79.7 | Accuracy (%) (macro-F1 mostly not reported), reasoning-required setting PQA-L test (500 questions) | Board | source |
| 16 | GPT-4 | 75.2 | Accuracy (%) (macro-F1 mostly not reported), reasoning-required setting PQA-L test (500 questions) | Board | source Zero-shot |
| — | MedGemma 27B (text-it) | 76.8 | Accuracy (%) (macro-F1 mostly not reported), reasoning-required setting PQA-L test (500 questions) | Self-reported | source Not on board; HF model card (Gemma 3 27B: 73.4). Training data include the PubMedQA train split. |
| — | Gemini 3 Pro | 82.2 | Accuracy (%) (macro-F1 mostly not reported), reasoning-required setting PQA-L test (500 questions) | Self-reported | source second-hand Not on board; from site note, not re-verified. |
Models reported
Grouped by relation type; each line shows its source type. Counted relations add up to 2. A pairing whose evidence rests only on this survey's own page is shown as unverified.
Developer report, official protocol (2)
- Frontier general-purpose LLMs Self-reported · Gemini 3 figures self-reported in the MedGemma 1.5 report; earlier board entries (GPT-4, Claude 3) were self-submitted. †partly verified · evidence: link
- MedGemma / Med-Gemini Self-reported · MedGemma 4B, 27B and 1.5 reported; Med-Gemini does not report PubMedQA. · evidence: link
Unconfirmed (6, not counted)
- Co-Scientist Unconfirmed source · No report found in official materials †disputed · evidence: link
- GPT-Rosalind Unconfirmed source · No report found †disputed · evidence: no source link
- Intern-S Unconfirmed source · No report found in official materials †disputed · evidence: link
- Quine Unconfirmed source · No report found in official materials †disputed · evidence: link
- SciReasoner Unconfirmed source · Not in the paper text †disputed · evidence: no source link
- VERID Unconfirmed source · No report found in official materials †disputed · evidence: no source link
Caveats
- The 500-question test set gives about ±1.9 points standard error, so top scores are indistinguishable; the leaderboard mixes prompting methods, and public data may be in training sets.
- The single-annotator human baseline (78.0) is a lower bound; most LLM entries report accuracy only, hiding whether the 'maybe' class is learned.
- MedGemma training data include the PubMedQA train split, so such scores are not contamination-free generalization.
- Aggregator figures of 87–95% for ensembles or knowledge-graph systems cannot be traced to primary sources and are not used.
- Tests reading comprehension of abstract conclusions only, not experimental design or evidence synthesis; no wet-lab translation evidence.
How to use it
Download PQA-L and evaluate on the 500-question test set; report accuracy and macro-F1, prompt template and setting, with bootstrap confidence intervals. Board entry requires sending predictions and a system description to the authors. Useful for coarse screening and small-model comparison, not for separating frontier models; pair with BioASQ and LitQA in LAB-Bench.
Pairs well with: BioASQ · LAB-Bench / LABBench2 · BixBench
Sources
- PubMedQA paper, arXiv 1909.06146 (preprint)
- EMNLP-IJCNLP 2019 version (paper)
- PubMedQA official leaderboard (leaderboard)
- PubMedQA data repository (repository)
- MedGemma technical report (developer) (preprint)
- MedGemma 27B model card (developer) (repository)
- MedGemma 1.5 report (includes Gemini 3 self-reported figures) (preprint)
Corrections
5 corrections
| Field | Was | Now | Basis | Found |
|---|---|---|---|---|
| first_release | 2019-08-23 | 2019-09-13 | source | 2026-11-16 |
| first_release_basis | GitHub repo initial commit (data pushed 2019-08-24); arXiv/EMNLP 2019 | arXiv v1 of the PubMedQA paper (2019-09-13); GitHub initial commit 2019-08-23 kept as a note only (repo creation excluded as evidence). | source | 2026-11-16 |
| latest_update_desc | README edit only; data frozen since 2019. | README edit on the data repository (2023-04-18); dataset itself frozen since 2019. 'Most recent sourced event' only - no claim that nothing happened later. | source | 2026-11-16 |
| latest_update_kind | none-since-release | code-maintenance-only | source | 2026-11-16 |
| status_note | Official leaderboard last updated November 2023 | Newest official-board entry dated 2024-04-18 | source | 2026-11-16 |
Part of the AI for Bio benchmarks survey · all 44 benchmarks · open in the explorer. Record as of 2026-11-16.