AI for Bio · Benchmark 14

PubMedQA

500-question yes/no/maybe test on PubMed abstracts; top scores now sit within sampling noise.

bench-14Type B+EEvidence L1Text, LLMs & agents○ Frozen Board中文

Overview

NamePubMedQA · aliases: PubMedQA, PQA-L, PQA-U, PQA-A
Full nameThe name has no expanded form.
Ambiguous names (not auto-linked)None
FamilyNot a family entry. Single static dataset; no family members.
DomainText, LLMs & agents
InclusionOriginal 30
TypeB E · Static text dataset, a routine LLM eval row
Evidence levelL1
PublisherUniversity of Pittsburgh, Carnegie Mellon University (Academic lab)
First release · paper v1
Basis: arXiv v1 of the PubMedQA paper (2019-09-13); GitHub initial commit 2019-08-23 kept as a note only (repo creation excluded as evidence). source
Other milestones
  • First public evidence: c. 2019 † · GitHub repository initial commit (repository event, excluded as first-release evidence) source
  • Preprint v1: · arXiv v1 of the PubMedQA paper (2019-09-13); GitHub initial commit 2019-08-23 kept as a note only (repo creation excluded as evidence). source
  • Journal online: † · EMNLP-IJCNLP 2019 conference paper source
Latest update · maintenance only
Basis: README commit on the data repository
README edit on the data repository (2023-04-18); dataset frozen since 2019. Most recent sourced event, not a claim that nothing happened later. source
The most recent sourced event, not a claim that nothing changed since.
Next expectedDataset frozen; nothing announced by the authors.
Status○ Frozen · Saturated; newest official-board entry dated 2024-04-18
ScalePQA-L 1.0k expert-labeled (500 test), PQA-U 61.2k, PQA-A 211.3k.
  • Expert-labelled questions (PQA-L): 1,000 (PQA-L)
  • Test set: 500 questions (PQA-L)
  • Unlabelled questions (PQA-U): 61.2k (PQA-U)
  • Auto-generated questions (PQA-A): 211.3k (PQA-A)
  • Label split yes/no/maybe: 55.2% / 33.8% / 11.0% (PQA-L)
Wet-lab refreshStatic data · None; fixed expert-labelled text data.
Own wet lab needed to evaluate a new modelNo, run it as a download
Models reported2 counted (self 2, unconfirmed 6)
Unverified fields† release_milestones.first_public_evidence, ranking.entries[5], models_reported[0]
As of2026-11-16

What it measures

  • Task: Answering yes/no/maybe questions from PubMed abstracts, testing reading comprehension of study conclusions.
  • Truth: Two annotators' classification of the authors' conclusion (PQA-L); authors estimate about 1% residual error.
  • Primary metric: Accuracy (reasoning-required setting)
  • Secondary metrics: Macro-F1
  • Version traps: State setting (reasoning-required or free), zero/few-shot, fine-tuning, retrieval or ensembling, and whether the train split was used.

Current ranking

Board

MetricAccuracy (%) (macro-F1 mostly not reported), reasoning-required setting
Eval setPQA-L test (500 questions)
Snapshot stale
Boardhttps://pubmedqa.github.io
Retrieved2026-11-16

Caveat. Newest board entry 2024-04-18; no later frontier reasoning models. Mixes zero-shot, few-shot, fine-tuned and ensembles, all self-submitted; top five within ~1.4 points vs ~±1.9-point binomial SE on 500 items.

Shown in the overview table:

82.0
Accuracy (%) (macro-F1 mostly not reported), reasoning-required setting · PQA-L test (500 questions)
Board · stale

RankModelScore (verbatim)Metric · eval set / protocolSource type · dateSource and note
1GPT-4 (Medprompt)82.0Accuracy (%) (macro-F1 mostly not reported), reasoning-required setting
PQA-L test (500 questions)
Board
source Overall top-1; prompting plus ensembling, self-submitted.
6GPT-4-base80.4Accuracy (%) (macro-F1 mostly not reported), reasoning-required setting
PQA-L test (500 questions)
Board
source
7Claude 379.7Accuracy (%) (macro-F1 mostly not reported), reasoning-required setting
PQA-L test (500 questions)
Board
source
16GPT-475.2Accuracy (%) (macro-F1 mostly not reported), reasoning-required setting
PQA-L test (500 questions)
Board
source Zero-shot
—MedGemma 27B (text-it)76.8Accuracy (%) (macro-F1 mostly not reported), reasoning-required setting
PQA-L test (500 questions)
Self-reported
source Not on board; HF model card (Gemma 3 27B: 73.4). Training data include the PubMedQA train split.
—Gemini 3 Pro82.2Accuracy (%) (macro-F1 mostly not reported), reasoning-required setting
PQA-L test (500 questions)
Self-reported
source second-hand Not on board; from site note, not re-verified.

Models reported

Grouped by relation type; each line shows its source type. Counted relations add up to 2. A pairing whose evidence rests only on this survey's own page is shown as unverified.

Developer report, official protocol (2)

Unconfirmed (6, not counted)

  • Co-Scientist Unconfirmed source · No report found in official materials †disputed · evidence: link
  • GPT-Rosalind Unconfirmed source · No report found †disputed · evidence: no source link
  • Intern-S Unconfirmed source · No report found in official materials †disputed · evidence: link
  • Quine Unconfirmed source · No report found in official materials †disputed · evidence: link
  • SciReasoner Unconfirmed source · Not in the paper text †disputed · evidence: no source link
  • VERID Unconfirmed source · No report found in official materials †disputed · evidence: no source link

Caveats

  1. The 500-question test set gives about ±1.9 points standard error, so top scores are indistinguishable; the leaderboard mixes prompting methods, and public data may be in training sets.
  2. The single-annotator human baseline (78.0) is a lower bound; most LLM entries report accuracy only, hiding whether the 'maybe' class is learned.
  3. MedGemma training data include the PubMedQA train split, so such scores are not contamination-free generalization.
  4. Aggregator figures of 87–95% for ensembles or knowledge-graph systems cannot be traced to primary sources and are not used.
  5. Tests reading comprehension of abstract conclusions only, not experimental design or evidence synthesis; no wet-lab translation evidence.

How to use it

Download PQA-L and evaluate on the 500-question test set; report accuracy and macro-F1, prompt template and setting, with bootstrap confidence intervals. Board entry requires sending predictions and a system description to the authors. Useful for coarse screening and small-model comparison, not for separating frontier models; pair with BioASQ and LitQA in LAB-Bench.

Pairs well with: BioASQ · LAB-Bench / LABBench2 · BixBench

Sources

  1. PubMedQA paper, arXiv 1909.06146 (preprint)
  2. EMNLP-IJCNLP 2019 version (paper)
  3. PubMedQA official leaderboard (leaderboard)
  4. PubMedQA data repository (repository)
  5. MedGemma technical report (developer) (preprint)
  6. MedGemma 27B model card (developer) (repository)
  7. MedGemma 1.5 report (includes Gemini 3 self-reported figures) (preprint)

Corrections

5 corrections
FieldWasNowBasisFound
first_release2019-08-232019-09-13source2026-11-16
first_release_basisGitHub repo initial commit (data pushed 2019-08-24); arXiv/EMNLP 2019arXiv v1 of the PubMedQA paper (2019-09-13); GitHub initial commit 2019-08-23 kept as a note only (repo creation excluded as evidence).source2026-11-16
latest_update_descREADME edit only; data frozen since 2019.README edit on the data repository (2023-04-18); dataset itself frozen since 2019. 'Most recent sourced event' only - no claim that nothing happened later.source2026-11-16
latest_update_kindnone-since-releasecode-maintenance-onlysource2026-11-16
status_noteOfficial leaderboard last updated November 2023Newest official-board entry dated 2024-04-18source2026-11-16

Part of the AI for Bio benchmarks survey · all 44 benchmarks · open in the explorer. Record as of 2026-11-16.