AI for Bio · Benchmark 42

BioMysteryBench

Questions on real sequencing data with metadata stripped, graded on the final answer only; authored by a model vendor, with only five public questions.

bench-42Type EEvidence L2–L4Text, LLMs & agents● Active Third-party中文

Overview

NameBioMysteryBench · aliases: BioMysteryBench, BioMysteryBench-preview
Full nameThe name is the full name; sources give no expanded form.
Ambiguous names (not auto-linked)None
FamilyNot a family entry. One question set; the preview is its public subset, not a family.
DomainText, LLMs & agents (also: Genomics & RNA)
InclusionSupplementary entry
TypeE · Agent question set with objectively verifiable answers
Evidence levelL2–L4 · Fixed set with objective truth; the full set is gated, close to a hidden-set practice but with no time-blind element
PublisherAnthropic (Big-tech lab)
First release
Basis: The publisher's research page, same day as the preview dataset's initial commit (2026-04-29) source
Other milestones
  • First public evidence: · Preview dataset initial commit, same day as the research page source
  • Official announcement: · Publisher research page source
Latest update† partly verified · board entry
Basis: Update date stated on the third-party leaderboard page
Third-party cross-vendor leaderboard updated (31 models, top 83.70%); the latest dataset change is preview v11 on 2026-07-07, swapping two questions.
Note: The leaderboard date is carried from an existing record and was not independently re-verified. source
The most recent sourced event, not a claim that nothing changed since.
Next expectedThe publisher has announced no next version or new round.
Status● Active · Active: third-party leaderboard updated 2026-11-09; dataset v11 (2026-07).
ScaleResearch page: 99 questions (76 human-solvable, 23 human-unsolved). Dataset card: 90 in the full set, 5 in the public preview.
  • Questions (research page): 99 (2026-04-29)
  • Human-solvable / human-unsolved: 76 / 23 (2026-04-29)
  • Questions (dataset card): 90 (preview v11, 2026-07-07)
  • Public preview questions: 5 (preview v11, 2026-07-07)
  • Dropped for formatting errors: 4 (2026-04-29)
Wet-lab refreshStatic data · No new wet lab; questions use existing sequencing data and answers are verified by orthogonal experiments or metadata.
Own wet lab needed to evaluate a new modelNo, run it as a download
Models reported1 counted (self 1, unconfirmed 6)
Unverified fields† latest_update, ranking.snapshot_date, scale.items[2], models_reported[model-gpt-rosalind], models_reported[model-scireasoner], models_reported[model-verid], models_reported[model-co-scientist], models_reported[model-intern-s], models_reported[model-quine]
As of2026-11-16

What it measures

  • Task: Given sequencing data with metadata stripped, the agent computationally recovers the hidden property; only the final answer is graded.
  • Truth: Objective properties verifiable from orthogonal experiments or the original metadata.
  • Primary metric: Final-answer accuracy
  • Secondary metrics: Stability of solutions across repeated runs; Separate scores on the human-solvable and human-unsolved subsets
  • Version traps: State whether a number uses the 99-question or 90-question set, the number of runs, the harness, the turn and time limits, and the judge model.

Current ranking

Third-party

MetricFinal-answer accuracy, mean of three full runs, scored by a judge model
Eval setThird-party board run: Terminus 2 harness, 200 turns / 4 h per task
Snapshot† partly verified
Boardhttps://vals.ai/benchmarks/biomysterybench
Retrieved2026-11-16

Caveat. The board is third-party run (31 models, 2026-11-09); the benchmark is vendor-built, the full set is gated, and vendor self-reported numbers are not comparable with the board.

Shown in the overview table:

83.70%
Final-answer accuracy, mean of three full runs, scored by a judge model · Third-party board run: Terminus 2 harness, 200 turns / 4 h per task
Third-party · †

RankModelScore (verbatim)Metric · eval set / protocolSource type · dateSource and note
1Claude Fable 5.283.70%Final-answer accuracy, mean of three full runs, scored by a judge model
Third-party board run: Terminus 2 harness, 200 turns / 4 h per task
Third-party
†
source Overall top-1.
2Claude Sonnet 5.581.11%Final-answer accuracy, mean of three full runs, scored by a judge model
Third-party board run: Terminus 2 harness, 200 turns / 4 h per task
Third-party
†
source
3GPT-6.1 Sol79.63%Final-answer accuracy, mean of three full runs, scored by a judge model
Third-party board run: Terminus 2 harness, 200 turns / 4 h per task
Third-party
†
source
4GPT-6 Astra79.26%Final-answer accuracy, mean of three full runs, scored by a judge model
Third-party board run: Terminus 2 harness, 200 turns / 4 h per task
Third-party
†
source Tied at 79.26% with two other entries.
7Gemini 4 Argon76.30%Final-answer accuracy, mean of three full runs, scored by a judge model
Third-party board run: Terminus 2 harness, 200 turns / 4 h per task
Third-party
†
source That vendor's highest entry.
—Claude Opus 4.677.4% (human-solvable) / 23.5% (human-unsolved)Final-answer accuracy, mean of five runs per question, vendor protocol
Publisher research page: 5 runs per question; human-solvable and human-unsolved subsets reported separately
Self-reported
source Vendor self-report with run settings different from the board; not ranked.

Models reported

Grouped by relation type; each line shows its source type. Counted relations add up to 1. A pairing whose evidence rests only on this survey's own page is shown as unverified.

Developer report, official protocol (1)

  • Frontier general-purpose LLMs Self-reported · The publisher runs the benchmark itself and reports scores; the cross-vendor board is run by a third party. · evidence: link

Unconfirmed (6, not counted)

  • Co-Scientist Unconfirmed source · No report on this benchmark found in official materials. †disputed · evidence: link
  • GPT-Rosalind Unconfirmed source · No report found; the only evidence is a site page, so it is shown as unconfirmed. †disputed · evidence: no source link
  • Intern-S Unconfirmed source · No report on this benchmark found in official materials. †disputed · evidence: link
  • Quine Unconfirmed source · No report on this benchmark found in official materials. †disputed · evidence: link
  • SciReasoner Unconfirmed source · The paper text does not mention this benchmark; the only evidence is a site page. †disputed · evidence: no source link
  • VERID Unconfirmed source · No report found; the only evidence is a site page, so it is shown as unconfirmed. †disputed · evidence: no source link

Caveats

  1. The questions were written by a model vendor that also develops models under test, which is a conflict of interest.
  2. The full set is gated and the public preview has only five questions, so outsiders cannot fully reproduce results.
  3. Question counts differ (99 vs 90) with no full public explanation, so cite the source.
  4. The sample is small with wide intervals, and success on hard questions is unstable, with marginal solves clearly more frequent.
  5. The vendor self-report and the third-party board use different run settings, so their numbers are not directly comparable.

How to use it

The five public preview questions support only a smoke test; the full set requires a request to the publisher under evaluation-only terms, or submission to the third-party board. Run each question several times and average, state the harness, turn and time limits and judge model, and report the human-solvable and human-unsolved subsets separately. Looking up source data by accession is prohibited. The sample is small, so gaps of a few points are not a ranking.

Pairs well with: LAB-Bench / LABBench2 · BixBench · BioML-bench

Sources

  1. Publisher research page (company blog) ·
  2. Preview dataset card (official site) ·
  3. Third-party cross-vendor leaderboard (leaderboard) ·

Corrections

2 corrections
FieldWasNowBasisFound
scale99 expert questions presented as the set sizeresearch page 99 (76 human-solvable, 23 human-unsolved) and dataset card 90 recorded side by side, with the public preview at 5source2026-11-16
latest_update.verificationtreated as fully verifiedpartial: the 2026-11-09 board date is carried from an existing record and was not independently re-fetchedsource2026-11-16

Part of the AI for Bio benchmarks survey · all 44 benchmarks · open in the explorer. Record as of 2026-11-16.