AI for Bio · Benchmark 18

LAB-Bench / LABBench2

Biology research-skill benchmark for LLMs and agents; the multiple-choice original is partly saturated and superseded by LABBench2.

bench-18Type EEvidence L1–L2Text, LLMs & agents● Active Board中文

Overview

NameLAB-Bench / LABBench2 · aliases: LAB-Bench, LABBench2, LitQA2, LitQA3, FigQA, ProtocolQA, SeqQA, SeqQA2, CloningScenarios, CloningQA, DbQA, SuppQA, TableQA
Full nameLanguage Agent Biology Benchmark
Ambiguous names (not auto-linked)LitQA
FamilyFamily entry: original LAB-Bench (multiple choice) and LABBench2 (open answers, tools allowed); question sets and scores are not comparable.
  • LAB-Bench ·
  • LABBench2 · †
DomainText, LLMs & agents
InclusionOriginal 30
TypeE · Research-skill question suites; LABBench2 adds executable verifiers
Evidence levelL1–L2 · CloningScenarios near L2
PublisherFutureHouse, Edison Scientific (Mixed)
First release · paper v1
Basis: LAB-Bench arXiv v1 (2407.10362) source
Other milestones
  • Preprint v1: · LAB-Bench arXiv v1 (2407.10362) source
Latest update · maintenance only
Basis: labbench2 repository commit
labbench2 harness commit: grading seqqa2 genes from annotated CDS files; several grader and runner fixes in October 2026. source
The most recent sourced event, not a claim that nothing changed since.
Next expectedNo next version or board update announced.
Status● Active · LABBench2 has an official board (updated 2026-03-23); original subsets partly saturated
ScaleOriginal: 2,457 questions, 30 subtasks; LABBench2: 1,912 tasks, 11 categories, no multiple choice.
  • Questions (paper Table 1): 2,457 (LAB-Bench)
  • Subtasks: 30 (LAB-Bench)
  • Public HF rows (about 20% held out): 1,967 (LAB-Bench)
  • Tasks (paper Table 1; release page says 1,892): 1,912 (LABBench2)
  • Categories: 11 (LABBench2)
Wet-lab refreshStatic data · None; questions written in-house and revised by version.
Own wet lab needed to evaluate a new modelNo, run it as a download
Models reported3 counted (board 1, modified protocol 2, unconfirmed 5)
Unverified fields† family.members[1].first_release, ranking.entries[5], models_reported[1], models_reported[2]
As of2026-11-16

What it measures

  • Task: Biology research skills of LLMs and agents: literature, databases, figures, sequences, cloning and protocol troubleshooting.
  • Truth: Expert-written answers, partly programmatic (ProtocolQA, SeqQA); LABBench2 sequence and cloning tasks use executable verifiers.
  • Primary metric: Accuracy (LABBench2: verifier or expert grading)
  • Secondary metrics: Precision (original); Coverage (original)
  • Version traps: Original and LABBench2 differ and cannot share a table; state version, tool use, inline context and board date; vendor system-card subitems are often derived variants.

Current ranking

Board

MetricLABBench2 aggregate score (mean across sub-benchmarks), settings 'Tools + high' and 'Base'
Eval setLABBench2 (1,912 tasks, 11 categories)
Snapshot
Boardhttps://advances.edisonscientific.com/benchmarks/labbench2/
Retrieved2026-11-16

Caveat. Board lists only 5 models (released 2026-02-05, updated 2026-03-23), no GPT-Rosalind or agent systems. The original has no board; vendor system-card numbers are self-reported subsets.

Shown in the overview table:

0.609 (Tools + high); Base 0.345
LABBench2 aggregate score (mean across sub-benchmarks), settings 'Tools + high' and 'Base' · LABBench2 (1,912 tasks, 11 categories)
Board ·

RankModelScore (verbatim)Metric · eval set / protocolSource type · dateSource and note
1gpt-5-20.609 (Tools + high); Base 0.345LABBench2 aggregate score (mean across sub-benchmarks), settings 'Tools + high' and 'Base'
LABBench2 (1,912 tasks, 11 categories)
Board
source Overall top-1 (Tools + high); Base top-1 is gemini-3-pro-preview (0.404).
2gpt-5-2-pro0.583 (Tools + high); Base 0.382LABBench2 aggregate score (mean across sub-benchmarks), settings 'Tools + high' and 'Base'
LABBench2 (1,912 tasks, 11 categories)
Board
source
3gemini-3-pro-preview0.564 (Tools + high); Base 0.404LABBench2 aggregate score (mean across sub-benchmarks), settings 'Tools + high' and 'Base'
LABBench2 (1,912 tasks, 11 categories)
Board
source
4claude-opus-4-60.519 (Tools + high); Base 0.349LABBench2 aggregate score (mean across sub-benchmarks), settings 'Tools + high' and 'Base'
LABBench2 (1,912 tasks, 11 categories)
Board
source
5claude-opus-4-50.471 (Tools + high); Base 0.325LABBench2 aggregate score (mean across sub-benchmarks), settings 'Tools + high' and 'Base'
LABBench2 (1,912 tasks, 11 categories)
Board
source
—BiomniWithheld: Incomplete provenance (missing url and date): the honesty rule forbids displaying this number.LABBench2 aggregate score (mean across sub-benchmarks), settings 'Tools + high' and 'Base'
Original LAB-Bench DbQA/SeqQA subsets (not LABBench2)
Self-reported
No sourced date†
no source link second-hand Original LAB-Bench subsets, not LABBench2; from site note (second-hand), not re-verified; not comparable to the board.

Models reported

Grouped by relation type; each line shows its source type. Counted relations add up to 3. A pairing whose evidence rests only on this survey's own page is shown as unverified.

Official leaderboard or results (1)

Developer report, modified protocol (2)

  • Biomni Self-reported LAB-Bench DbQA/SeqQA subsets · Original DbQA/SeqQA subsets with 45 dev / 315 hidden test questions and 3 runs. low confidence †unverified · evidence: link
  • GPT-Rosalind Self-reported · 11 of 15 LABBench2 tasks, qualitative only. †partly verified · evidence: link

Unconfirmed (5, not counted)

  • Co-Scientist Unconfirmed source · No report found in official materials †disputed · evidence: link
  • Intern-S Unconfirmed source · No report found in official materials †disputed · evidence: link
  • Quine Unconfirmed source · No report found in official materials †disputed · evidence: link
  • SciReasoner Unconfirmed source · Not in the paper text †disputed · evidence: no source link
  • VERID Unconfirmed source · No report found in official materials †disputed · evidence: no source link

Caveats

  1. Multiple choice inflated original scores; small subsets (41 and 135 questions) are noisy; the public 80% may be contaminated; no wet-lab evidence.
  2. The LABBench2 paper reports accuracy 26–46 points below the original; original scores do not represent open-answer ability.
  3. Vendor system-card subitems (e.g. open-ended ProtocolQA) are derived variants, and the same subitem circulates with several numbers; use the primary document.
  4. ProtocolQA is paper-based error finding and involves no actual experiment.

How to use it

Run from the repository or Hugging Face data; no wet lab needed; the original needs an agent function returning the option letter. Prefer LABBench2 and state tool use (search, code execution), inline context and board date; the private 20% hold-out cannot be checked for contamination. Pair with BixBench and wet-lab blind tests such as Adaptyv.

Pairs well with: BixBench · Adaptyv competitions / BenchBB · PubMedQA · BioASQ

Sources

  1. LAB-Bench paper, arXiv 2407.10362 (preprint)
  2. LABBench2 paper, arXiv 2604.09554 (preprint)
  3. LABBench2 official board (leaderboard)
  4. LABBench2 release page (company blog)
  5. labbench2 repository (repository)
  6. LAB-Bench repository (repository)
  7. LAB-Bench on Hugging Face (repository)
  8. Claude Opus 4.5 system card (vendor self-report) (company blog)
  9. Biomni preprint (developer) (preprint)

Corrections

1 correction
FieldWasNowBasisFound
ranking.entries[Biomni]Biomni original LAB-Bench subset scores shown as a ranking entryScore withheld: entry lacks url and datesource2026-11-16

Part of the AI for Bio benchmarks survey · all 44 benchmarks · open in the explorer. Record as of 2026-11-16.