Overview
| Name | BioMysteryBench · aliases: BioMysteryBench, BioMysteryBench-preview |
|---|---|
| Full name | The name is the full name; sources give no expanded form. |
| Ambiguous names (not auto-linked) | None |
| Family | Not a family entry. One question set; the preview is its public subset, not a family. |
| Domain | Text, LLMs & agents (also: Genomics & RNA) |
| Inclusion | Supplementary entry |
| Type | E · Agent question set with objectively verifiable answers |
| Evidence level | L2–L4 · Fixed set with objective truth; the full set is gated, close to a hidden-set practice but with no time-blind element |
| Publisher | Anthropic (Big-tech lab) |
| First release | Basis: The publisher's research page, same day as the preview dataset's initial commit (2026-04-29) source |
| Other milestones | |
| Latest update | † partly verified · board entry Basis: Update date stated on the third-party leaderboard page Third-party cross-vendor leaderboard updated (31 models, top 83.70%); the latest dataset change is preview v11 on 2026-07-07, swapping two questions. Note: The leaderboard date is carried from an existing record and was not independently re-verified. source The most recent sourced event, not a claim that nothing changed since. |
| Next expected | The publisher has announced no next version or new round. |
| Status | ● Active · Active: third-party leaderboard updated 2026-11-09; dataset v11 (2026-07). |
| Scale | Research page: 99 questions (76 human-solvable, 23 human-unsolved). Dataset card: 90 in the full set, 5 in the public preview.
|
| Wet-lab refresh | Static data · No new wet lab; questions use existing sequencing data and answers are verified by orthogonal experiments or metadata. |
| Own wet lab needed to evaluate a new model | No, run it as a download |
| Models reported | 1 counted (self 1, unconfirmed 6) |
| Unverified fields | † latest_update, ranking.snapshot_date, scale.items[2], models_reported[model-gpt-rosalind], models_reported[model-scireasoner], models_reported[model-verid], models_reported[model-co-scientist], models_reported[model-intern-s], models_reported[model-quine] |
| As of | 2026-11-16 |
What it measures
- Task: Given sequencing data with metadata stripped, the agent computationally recovers the hidden property; only the final answer is graded.
- Truth: Objective properties verifiable from orthogonal experiments or the original metadata.
- Primary metric: Final-answer accuracy
- Secondary metrics: Stability of solutions across repeated runs; Separate scores on the human-solvable and human-unsolved subsets
- Version traps: State whether a number uses the 99-question or 90-question set, the number of runs, the harness, the turn and time limits, and the judge model.
Current ranking
Third-party
| Metric | Final-answer accuracy, mean of three full runs, scored by a judge model |
|---|---|
| Eval set | Third-party board run: Terminus 2 harness, 200 turns / 4 h per task |
| Snapshot | † partly verified |
| Board | https://vals.ai/benchmarks/biomysterybench |
| Retrieved | 2026-11-16 |
Caveat. The board is third-party run (31 models, 2026-11-09); the benchmark is vendor-built, the full set is gated, and vendor self-reported numbers are not comparable with the board.
Shown in the overview table: Third-party
| Rank | Model | Score (verbatim) | Metric · eval set / protocol | Source type · date | Source and note |
|---|---|---|---|---|---|
| 1 | Claude Fable 5.2 | 83.70% | Final-answer accuracy, mean of three full runs, scored by a judge model Third-party board run: Terminus 2 harness, 200 turns / 4 h per task | Third-party † | source Overall top-1. |
| 2 | Claude Sonnet 5.5 | 81.11% | Final-answer accuracy, mean of three full runs, scored by a judge model Third-party board run: Terminus 2 harness, 200 turns / 4 h per task | Third-party † | source |
| 3 | GPT-6.1 Sol | 79.63% | Final-answer accuracy, mean of three full runs, scored by a judge model Third-party board run: Terminus 2 harness, 200 turns / 4 h per task | Third-party † | source |
| 4 | GPT-6 Astra | 79.26% | Final-answer accuracy, mean of three full runs, scored by a judge model Third-party board run: Terminus 2 harness, 200 turns / 4 h per task | Third-party † | source Tied at 79.26% with two other entries. |
| 7 | Gemini 4 Argon | 76.30% | Final-answer accuracy, mean of three full runs, scored by a judge model Third-party board run: Terminus 2 harness, 200 turns / 4 h per task | Third-party † | source That vendor's highest entry. |
| — | Claude Opus 4.6 | 77.4% (human-solvable) / 23.5% (human-unsolved) | Final-answer accuracy, mean of five runs per question, vendor protocol Publisher research page: 5 runs per question; human-solvable and human-unsolved subsets reported separately | Self-reported | source Vendor self-report with run settings different from the board; not ranked. |
Models reported
Grouped by relation type; each line shows its source type. Counted relations add up to 1. A pairing whose evidence rests only on this survey's own page is shown as unverified.
Developer report, official protocol (1)
- Frontier general-purpose LLMs Self-reported · The publisher runs the benchmark itself and reports scores; the cross-vendor board is run by a third party. · evidence: link
Unconfirmed (6, not counted)
- Co-Scientist Unconfirmed source · No report on this benchmark found in official materials. †disputed · evidence: link
- GPT-Rosalind Unconfirmed source · No report found; the only evidence is a site page, so it is shown as unconfirmed. †disputed · evidence: no source link
- Intern-S Unconfirmed source · No report on this benchmark found in official materials. †disputed · evidence: link
- Quine Unconfirmed source · No report on this benchmark found in official materials. †disputed · evidence: link
- SciReasoner Unconfirmed source · The paper text does not mention this benchmark; the only evidence is a site page. †disputed · evidence: no source link
- VERID Unconfirmed source · No report found; the only evidence is a site page, so it is shown as unconfirmed. †disputed · evidence: no source link
Caveats
- The questions were written by a model vendor that also develops models under test, which is a conflict of interest.
- The full set is gated and the public preview has only five questions, so outsiders cannot fully reproduce results.
- Question counts differ (99 vs 90) with no full public explanation, so cite the source.
- The sample is small with wide intervals, and success on hard questions is unstable, with marginal solves clearly more frequent.
- The vendor self-report and the third-party board use different run settings, so their numbers are not directly comparable.
How to use it
The five public preview questions support only a smoke test; the full set requires a request to the publisher under evaluation-only terms, or submission to the third-party board. Run each question several times and average, state the harness, turn and time limits and judge model, and report the human-solvable and human-unsolved subsets separately. Looking up source data by accession is prohibited. The sample is small, so gaps of a few points are not a ranking.
Pairs well with: LAB-Bench / LABBench2 · BixBench · BioML-bench
Sources
- Publisher research page (company blog) ·
- Preview dataset card (official site) ·
- Third-party cross-vendor leaderboard (leaderboard) ·
Corrections
2 corrections
| Field | Was | Now | Basis | Found |
|---|---|---|---|---|
| scale | 99 expert questions presented as the set size | research page 99 (76 human-solvable, 23 human-unsolved) and dataset card 90 recorded side by side, with the public preview at 5 | source | 2026-11-16 |
| latest_update.verification | treated as fully verified | partial: the 2026-11-09 board date is carried from an existing record and was not independently re-fetched | source | 2026-11-16 |
Part of the AI for Bio benchmarks survey · all 44 benchmarks · open in the explorer. Record as of 2026-11-16.