AI for Bio · Benchmark 23

BioASQ

The only yearly biomedical retrieval and QA challenge with new expert-written questions and expert grading

bench-23Type A+BEvidence L1–L4Text, LLMs & agents● Active Board中文

Overview

NameBioASQ · aliases: BioASQ, BioASQ 14, BioASQ 13, Task 14b, BioASQ-QA, Synergy task
Full nameBioASQ challenge on biomedical semantic indexing and question answering
Ambiguous names (not auto-linked)Task a, Task b
FamilyNot a family entry. Successive yearly editions of one challenge rather than a family entry; past questions are released separately as static datasets.
DomainText, LLMs & agents
InclusionOriginal 30
TypeA B · Yearly challenge whose past questions are public static sets
Evidence levelL1–L4 · Reproduces literature knowledge, while each new batch approximates a blind test
PublisherBioASQ consortium, National Centre for Scientific Research Demokritos, Athens University of Economics and Business (Academic lab)
First release† partly verified
Basis: First BioASQ challenge at the CLEF 2013 lab, with the EU project starting in October 2012; the month has no primary source source
Other milestones
  • Journal online: † · Results paper for the first challenge; year only source
  • Challenge opens: † · Earliest edition listed on the official workshop page source
Latest update · results
Basis: Publication of the BioASQ 14 overview paper in CEUR-WS Vol-4283 with preliminary Task 14b results
The BioASQ 14 overview paper appeared in CEUR Vol-4283 with preliminary Task 14b results; the tasks ran in spring 2026.
Note: The CLEF meeting dates of 2026-09-21 to 24 are a conference event, not a benchmark event. source
The most recent sourced event, not a claim that nothing changed since.
Next expectedNo BioASQ 15 announcement was found; the official site has only a placeholder page.
Status● Active · Active: the 14th edition published its overview and preliminary results in 2026-09
ScaleTask 14b: 5,729 training questions, 280 test questions in 4 batches, 151 systems and 821 submissions.
  • Training questions: 5729 (14b)
  • Test questions in 4 batches of 80/80/60/60: 280 (14b)
  • Systems: 151 (14b)
  • Submissions: 821 (14b)
  • Teams, of which 48 in the QA part: 87 (BioASQ 14)
Wet-lab refreshGrows from public sources · No: about 280 new expert-written questions a year, with expert-annotated literature as truth rather than wet-lab data.
Own wet lab needed to evaluate a new modelNo, run it as a download
Models reported0 counted (unconfirmed 8)
Unverified fields† first_release, release_milestones.challenge_open, release_milestones.journal_online, ranking.snapshot_date, ranking.entries[4]
As of2026-11-16

What it measures

  • Task: Task b runs in phases: retrieve PubMed documents and snippets, or answer given gold material; question types are yes/no, factoid, list and summary.
  • Truth: Expert-written questions with gold documents, snippets and answers, finalized after experts add missed items post-evaluation.
  • Primary metric: Yes/no macro-F1 and retrieval MAP/F1
  • Secondary metrics: Factoid MRR; List F1; Manual scores and ROUGE for summaries
  • Version traps: Scores depend on which experts wrote the questions, so years do not compare; state the edition, batch and phase (A, A+, B) and distinguish the BLURB yes/no slice.

Current ranking

Board

MetricYes/no macro-F1 as the ranking key of Tables 7 and 8, plus factoid MRR and list F1
Eval setBioASQ Task 14b batch 2 exact answers, phase A+ (Table 7) and phase B (Table 8); preliminary results
Snapshot† partly verified
Boardhttps://ceur-ws.org/Vol-4283/paper1.pdf
Retrieved2026-11-16

Caveat. The official board ranks submitted systems per batch rather than underlying models, and 14b results are preliminary until expert re-assessment; scores do not compare across years.

Shown in the overview table:

mckpt2
Yes/No F1 0.9481, Acc. 0.9524; Factoid MRR 0.2000; List F1 0.1537
Yes/no macro-F1 as the ranking key of Tables 7 and 8, plus factoid MRR and list F1 · BioASQ Task 14b batch 2 exact answers, phase A+ (Table 7) and phase B (Table 8); preliminary results
Board · †

RankModelScore (verbatim)Metric · eval set / protocolSource type · dateSource and note
1mckpt2Yes/No F1 0.9481, Acc. 0.9524; Factoid MRR 0.2000; List F1 0.1537Yes/no macro-F1 as the ranking key of Tables 7 and 8, plus factoid MRR and list F1
BioASQ Task 14b batch 2 exact answers, phase A+ (Table 7) and phase B (Table 8); preliminary results
Board
source Not a site model; top of Table 7 for phase A+ batch 2, tied on yes/no F1 with another system.
3MedQA-1 (university team; phase B uses a frontier-LLM ensemble)Yes/No F1 0.9443, Acc. 0.9524; Factoid MRR 0.4000; List F1 0.3551Yes/no macro-F1 as the ranking key of Tables 7 and 8, plus factoid MRR and list F1
BioASQ Task 14b batch 2 exact answers, phase A+ (Table 7) and phase B (Table 8); preliminary results
Board
source Third row of Table 7, phase A+ batch 2; 0.9443 is tied across many systems so the rank is only a listing position. The board ranks submitted systems and the model attribution comes from the system description.
1dictycite-baselineYes/No F1 1.0000, Acc. 1.0000; Factoid MRR 0.3000; List F1 0.4301Yes/no macro-F1 as the ranking key of Tables 7 and 8, plus factoid MRR and list F1
BioASQ Task 14b batch 2 exact answers, phase A+ (Table 7) and phase B (Table 8); preliminary results
Board
source Not a site model; top of Table 8 for phase B batch 2.
—MedQA-1Yes/No F1 0.9443, Acc. 0.9524; Factoid MRR 0.4000; List F1 0.4433Yes/no macro-F1 as the ranking key of Tables 7 and 8, plus factoid MRR and list F1
BioASQ Task 14b batch 2 exact answers, phase A+ (Table 7) and phase B (Table 8); preliminary results
Board
source Table 8, phase B batch 2; the yes/no F1 of 0.9443 is tied across many systems, so there is no unique rank.
—MedGemma-27B-IT as a component of participant systemsWithheld: No standalone model figure can be cited, so no score is shown.Yes/no macro-F1 as the ranking key of Tables 7 and 8, plus factoid MRR and list F1
BioASQ Task 14b batch 2 exact answers, phase A+ (Table 7) and phase B (Table 8); preliminary results
Third-party
source No standalone model score exists: the official board scores whole systems and the batch-2 indicative tables read here contain no such row.

Models reported

Grouped by relation type; each line shows its source type. Counted relations add up to 0. A pairing whose evidence rests only on this survey's own page is shown as unverified.

Unconfirmed (8, not counted)

  • Co-Scientist Unconfirmed source · No report by it was found in the official material. †disputed · evidence: link
  • Frontier general-purpose LLMs Unconfirmed source · The overview ranks submitted systems, not the underlying LLMs. †disputed · evidence: link
  • GPT-Rosalind Unconfirmed source · No report was found. †disputed · evidence: no source link
  • Intern-S Unconfirmed source · No report by it was found in the official material. †disputed · evidence: link
  • MedGemma / Med-Gemini Unconfirmed source · The overview scores whole systems and contains no standalone figure for it, so the board marking in the matrix is unsupported. †disputed · evidence: link
  • Quine Unconfirmed source · No report by it was found in the official material. †disputed · evidence: link
  • SciReasoner Unconfirmed source · Not found in the paper text. †disputed · evidence: no source link
  • VERID Unconfirmed source · No report by it was found in the official material. †disputed · evidence: no source link

Caveats

  1. Scores depend on the question writers: 64% of 14b questions came from new experts and the top list and factoid means fell below the previous edition, so years do not compare.
  2. Yes/no has only 11-21 questions per batch, so one question is worth 5-9 points, and the organizers give no confidence intervals or significance tests.
  3. Past questions are public and LLM training corpora very likely contain them, with the contamination unquantified.
  4. The board ranks whole systems including retrieval, reranking and ensembling, so single-model performance cannot be read off it.
  5. The corpus is PubMed abstracts rather than full texts, and with no wet-lab translation it only proxies literature skill.

How to use it

No wet lab needed. A strict comparison means submitting in the official window; offline, the newest batch can serve as an uncontaminated hold-out, but only for the current edition. Report phases A, A+ and B together, since phase B alone hides accumulated retrieval errors; registration is free and in-batch submission is time-limited.

Pairs well with: PubMedQA · LAB-Bench / LABBench2 · BioMysteryBench

Sources

  1. BioASQ 14 overview paper, CEUR Vol-4283 (paper) · †
  2. BioASQ 13 overview paper (preprint)
  3. BioASQ official site (official site)
  4. Results paper for the first challenge (journal)
  5. BioASQ-QA dataset paper (journal)
  6. Statement on the discontinuation of Task a (preprint)
  7. Third-party LLM evaluation on BioASQ (third-party paper)

Corrections

6 corrections
FieldWasNowBasisFound
latest_update2026-09-212026-09source2026-11-16
latest_update_descBioASQ 14 (CLEF 2026, Jena, 21-24 Sep) concluded; overview paper in CEUR-WS Vol-4283; six tasks incl. new MultiClinSum-2, BioNNE-R, ELCardioCC, GutBrainIE.BioASQ 14 (CLEF 2026) overview paper published in CEUR Vol-4283 with preliminary Task 14b results; tasks ran spring 2026. CLEF meeting (Jena, 2026-09-21..24) is a conference date, not a benchmark event.source2026-11-16
latest_update_kindnew-roundresults-publishedsource2026-11-16
ranking.entriesMedQA-1 list F1 0.3250.3551 (Table 7, phase A+, batch 2)source2026-11-16
models_reportedMedGemma / Med-Gemini marked as a board pairing in the matrixdisputed-or-unconfirmed: the overview scores whole systems and reports no standalone figuresource2026-11-16
models_reportedfrontier LLMs marked as a board pairingdisputed-or-unconfirmed: systems are ranked, not base LLMssource2026-11-16

Part of the AI for Bio benchmarks survey · all 44 benchmarks · open in the explorer. Record as of 2026-11-16.