Overview
| Name | BioASQ · aliases: BioASQ, BioASQ 14, BioASQ 13, Task 14b, BioASQ-QA, Synergy task |
|---|---|
| Full name | BioASQ challenge on biomedical semantic indexing and question answering |
| Ambiguous names (not auto-linked) | Task a, Task b |
| Family | Not a family entry. Successive yearly editions of one challenge rather than a family entry; past questions are released separately as static datasets. |
| Domain | Text, LLMs & agents |
| Inclusion | Original 30 |
| Type | A B · Yearly challenge whose past questions are public static sets |
| Evidence level | L1–L4 · Reproduces literature knowledge, while each new batch approximates a blind test |
| Publisher | BioASQ consortium, National Centre for Scientific Research Demokritos, Athens University of Economics and Business (Academic lab) |
| First release | † partly verified Basis: First BioASQ challenge at the CLEF 2013 lab, with the EU project starting in October 2012; the month has no primary source source |
| Other milestones | |
| Latest update | · results Basis: Publication of the BioASQ 14 overview paper in CEUR-WS Vol-4283 with preliminary Task 14b results The BioASQ 14 overview paper appeared in CEUR Vol-4283 with preliminary Task 14b results; the tasks ran in spring 2026. Note: The CLEF meeting dates of 2026-09-21 to 24 are a conference event, not a benchmark event. source The most recent sourced event, not a claim that nothing changed since. |
| Next expected | No BioASQ 15 announcement was found; the official site has only a placeholder page. |
| Status | ● Active · Active: the 14th edition published its overview and preliminary results in 2026-09 |
| Scale | Task 14b: 5,729 training questions, 280 test questions in 4 batches, 151 systems and 821 submissions.
|
| Wet-lab refresh | Grows from public sources · No: about 280 new expert-written questions a year, with expert-annotated literature as truth rather than wet-lab data. |
| Own wet lab needed to evaluate a new model | No, run it as a download |
| Models reported | 0 counted (unconfirmed 8) |
| Unverified fields | † first_release, release_milestones.challenge_open, release_milestones.journal_online, ranking.snapshot_date, ranking.entries[4] |
| As of | 2026-11-16 |
What it measures
- Task: Task b runs in phases: retrieve PubMed documents and snippets, or answer given gold material; question types are yes/no, factoid, list and summary.
- Truth: Expert-written questions with gold documents, snippets and answers, finalized after experts add missed items post-evaluation.
- Primary metric: Yes/no macro-F1 and retrieval MAP/F1
- Secondary metrics: Factoid MRR; List F1; Manual scores and ROUGE for summaries
- Version traps: Scores depend on which experts wrote the questions, so years do not compare; state the edition, batch and phase (A, A+, B) and distinguish the BLURB yes/no slice.
Current ranking
Board
| Metric | Yes/no macro-F1 as the ranking key of Tables 7 and 8, plus factoid MRR and list F1 |
|---|---|
| Eval set | BioASQ Task 14b batch 2 exact answers, phase A+ (Table 7) and phase B (Table 8); preliminary results |
| Snapshot | † partly verified |
| Board | https://ceur-ws.org/Vol-4283/paper1.pdf |
| Retrieved | 2026-11-16 |
Caveat. The official board ranks submitted systems per batch rather than underlying models, and 14b results are preliminary until expert re-assessment; scores do not compare across years.
Shown in the overview table: Board
| Rank | Model | Score (verbatim) | Metric · eval set / protocol | Source type · date | Source and note |
|---|---|---|---|---|---|
| 1 | mckpt2 | Yes/No F1 0.9481, Acc. 0.9524; Factoid MRR 0.2000; List F1 0.1537 | Yes/no macro-F1 as the ranking key of Tables 7 and 8, plus factoid MRR and list F1 BioASQ Task 14b batch 2 exact answers, phase A+ (Table 7) and phase B (Table 8); preliminary results | Board | source Not a site model; top of Table 7 for phase A+ batch 2, tied on yes/no F1 with another system. |
| 3 | MedQA-1 (university team; phase B uses a frontier-LLM ensemble) | Yes/No F1 0.9443, Acc. 0.9524; Factoid MRR 0.4000; List F1 0.3551 | Yes/no macro-F1 as the ranking key of Tables 7 and 8, plus factoid MRR and list F1 BioASQ Task 14b batch 2 exact answers, phase A+ (Table 7) and phase B (Table 8); preliminary results | Board | source Third row of Table 7, phase A+ batch 2; 0.9443 is tied across many systems so the rank is only a listing position. The board ranks submitted systems and the model attribution comes from the system description. |
| 1 | dictycite-baseline | Yes/No F1 1.0000, Acc. 1.0000; Factoid MRR 0.3000; List F1 0.4301 | Yes/no macro-F1 as the ranking key of Tables 7 and 8, plus factoid MRR and list F1 BioASQ Task 14b batch 2 exact answers, phase A+ (Table 7) and phase B (Table 8); preliminary results | Board | source Not a site model; top of Table 8 for phase B batch 2. |
| — | MedQA-1 | Yes/No F1 0.9443, Acc. 0.9524; Factoid MRR 0.4000; List F1 0.4433 | Yes/no macro-F1 as the ranking key of Tables 7 and 8, plus factoid MRR and list F1 BioASQ Task 14b batch 2 exact answers, phase A+ (Table 7) and phase B (Table 8); preliminary results | Board | source Table 8, phase B batch 2; the yes/no F1 of 0.9443 is tied across many systems, so there is no unique rank. |
| — | MedGemma-27B-IT as a component of participant systems | Withheld: No standalone model figure can be cited, so no score is shown. | Yes/no macro-F1 as the ranking key of Tables 7 and 8, plus factoid MRR and list F1 BioASQ Task 14b batch 2 exact answers, phase A+ (Table 7) and phase B (Table 8); preliminary results | Third-party | source No standalone model score exists: the official board scores whole systems and the batch-2 indicative tables read here contain no such row. |
Models reported
Grouped by relation type; each line shows its source type. Counted relations add up to 0. A pairing whose evidence rests only on this survey's own page is shown as unverified.
Unconfirmed (8, not counted)
- Co-Scientist Unconfirmed source · No report by it was found in the official material. †disputed · evidence: link
- Frontier general-purpose LLMs Unconfirmed source · The overview ranks submitted systems, not the underlying LLMs. †disputed · evidence: link
- GPT-Rosalind Unconfirmed source · No report was found. †disputed · evidence: no source link
- Intern-S Unconfirmed source · No report by it was found in the official material. †disputed · evidence: link
- MedGemma / Med-Gemini Unconfirmed source · The overview scores whole systems and contains no standalone figure for it, so the board marking in the matrix is unsupported. †disputed · evidence: link
- Quine Unconfirmed source · No report by it was found in the official material. †disputed · evidence: link
- SciReasoner Unconfirmed source · Not found in the paper text. †disputed · evidence: no source link
- VERID Unconfirmed source · No report by it was found in the official material. †disputed · evidence: no source link
Caveats
- Scores depend on the question writers: 64% of 14b questions came from new experts and the top list and factoid means fell below the previous edition, so years do not compare.
- Yes/no has only 11-21 questions per batch, so one question is worth 5-9 points, and the organizers give no confidence intervals or significance tests.
- Past questions are public and LLM training corpora very likely contain them, with the contamination unquantified.
- The board ranks whole systems including retrieval, reranking and ensembling, so single-model performance cannot be read off it.
- The corpus is PubMed abstracts rather than full texts, and with no wet-lab translation it only proxies literature skill.
How to use it
No wet lab needed. A strict comparison means submitting in the official window; offline, the newest batch can serve as an uncontaminated hold-out, but only for the current edition. Report phases A, A+ and B together, since phase B alone hides accumulated retrieval errors; registration is free and in-batch submission is time-limited.
Pairs well with: PubMedQA · LAB-Bench / LABBench2 · BioMysteryBench
Sources
- BioASQ 14 overview paper, CEUR Vol-4283 (paper) · †
- BioASQ 13 overview paper (preprint)
- BioASQ official site (official site)
- Results paper for the first challenge (journal)
- BioASQ-QA dataset paper (journal)
- Statement on the discontinuation of Task a (preprint)
- Third-party LLM evaluation on BioASQ (third-party paper)
Corrections
6 corrections
| Field | Was | Now | Basis | Found |
|---|---|---|---|---|
| latest_update | 2026-09-21 | 2026-09 | source | 2026-11-16 |
| latest_update_desc | BioASQ 14 (CLEF 2026, Jena, 21-24 Sep) concluded; overview paper in CEUR-WS Vol-4283; six tasks incl. new MultiClinSum-2, BioNNE-R, ELCardioCC, GutBrainIE. | BioASQ 14 (CLEF 2026) overview paper published in CEUR Vol-4283 with preliminary Task 14b results; tasks ran spring 2026. CLEF meeting (Jena, 2026-09-21..24) is a conference date, not a benchmark event. | source | 2026-11-16 |
| latest_update_kind | new-round | results-published | source | 2026-11-16 |
| ranking.entries | MedQA-1 list F1 0.325 | 0.3551 (Table 7, phase A+, batch 2) | source | 2026-11-16 |
| models_reported | MedGemma / Med-Gemini marked as a board pairing in the matrix | disputed-or-unconfirmed: the overview scores whole systems and reports no standalone figure | source | 2026-11-16 |
| models_reported | frontier LLMs marked as a board pairing | disputed-or-unconfirmed: systems are ranked, not base LLMs | source | 2026-11-16 |
Part of the AI for Bio benchmarks survey · all 44 benchmarks · open in the explorer. Record as of 2026-11-16.