AI for Bio · Benchmark 41

BioML-bench

Agents build and run biomedical ML pipelines end to end, scored against human public-leaderboard results; tested agents average below the human baseline.

bench-41Type EEvidence L2Text, LLMs & agents◐ Low-frequency Self-reported中文

Overview

NameBioML-bench · aliases: BioML-bench, BioML-Bench, biomlbench
Full nameThe name is the full name; sources give no expanded form.
Ambiguous names (not auto-linked)None
FamilyNot a family entry. One task set with no family members; BioXArena is a separate follow-up benchmark.
DomainText, LLMs & agents
InclusionSupplementary entry
TypeE · End-to-end agent task environment
Evidence levelL2 · Truth hidden from the agent, but data come from already public competitions
PublisherScienceMachine, University of Cambridge, Shift Bioscience, University of Toronto, Vector Institute (Mixed)
First release · paper v1
Basis: bioRxiv preprint v1 (the 2025-05-19 first repository commit predates the paper and is a note only) source
Other milestones
  • First public evidence: · Earliest sourced public evidence is preprint v1; repository creation and first commit are not used as a first release source
  • Preprint v1: · bioRxiv v1 (DOI 10.1101/2025.09.01.673319) source
Latest update · paper revision
Basis: bioRxiv version metadata
Preprint v3; last repository commit 2025-09-14 (documentation fixes), maintenance only since. source
The most recent sourced event, not a claim that nothing changed since.
Next expectedThe maintainers have announced no next version or new round.
Status◐ Low-frequency · Low frequency: no new version after preprint v3; independent follow-up use in 2026.
ScaleFour domains; tasks taken from public competitions and datasets, with a follow-up report citing 24 tasks.
  • Domains: 4 (preprint v3)
  • Task sources: PolarisHub, Kaggle, ProteinGym, OpenProblems (preprint v3)
  • Tasks (follow-up report): 24 (third-party report 2026-05)
  • Runs per agent per task: 4 (preprint v3)
  • Failed runs: 41 (preprint v3)
Wet-lab refreshStatic data · Produces no new experimental data; tasks and truth come from existing public competitions and datasets.
Own wet lab needed to evaluate a new modelNo, run it as a download
Models reported1 counted (retrospective 1, unconfirmed 6)
Unverified fields† ranking.entries[0].date, scale.items[2], models_reported[model-gpt-rosalind], models_reported[model-scireasoner], models_reported[model-verid], models_reported[model-co-scientist], models_reported[model-intern-s], models_reported[model-quine]
As of2026-11-16

What it measures

  • Task: Given a task description, training data and a submission example, the agent builds a model and submits predictions.
  • Truth: Hidden test labels from public competitions and datasets; human baselines come from public leaderboards.
  • Primary metric: Percentile against human leaderboards, with failed runs penalized
  • Secondary metrics: Each task's native metric (AUROC, Spearman and others); Mean rank; Number of failed runs
  • Version traps: State whether failure penalties are applied, which task-set version, which LLM backbone, and the budget and time limits.

Current ranking

Self-reported

MetricMean leaderboard percentile against human public-leaderboard baselines, failed runs penalized
Eval setBioML-bench tasks across 4 domains (24 tasks in the follow-up report), 4 runs per agent per task
Snapshot
BoardNo official board
Retrieved2026-11-16

Caveat. No official live leaderboard; numbers come from different papers and protocols and are not ranked against each other. Several agent models have no sourced number here.

No headline entry: The source has no rank-1 entry: entries are unranked or protocol-specific, so the caveat is shown as for a not-comparable cell.

RankModelScore (verbatim)Metric · eval set / protocolSource type · dateSource and note
—Biomni34.45 ± 5.36 (leaderboard percentile)Mean leaderboard percentile against human public-leaderboard baselines, failed runs penalized
BioML-bench tasks across 4 domains (24 tasks in the follow-up report), 4 runs per agent per task
Self-reported
†
source Best of the four agents re-run by the benchmark authors; mean rank 1.88 +/- 0.18, no failed tasks.
—AutoScientists74.4% (mean leaderboard percentile, 24 tasks)Mean leaderboard percentile against human public-leaderboard baselines, failed runs penalized
BioML-bench tasks across 4 domains (24 tasks in the follow-up report), 4 runs per agent per task
Self-reported
source Not a site model; developer-reported with partly different hardware and protocol, and the report does not say whether the protocol matches the row above.

Models reported

Grouped by relation type; each line shows its source type. Counted relations add up to 1. A pairing whose evidence rests only on this survey's own page is shown as unverified.

Retrospective dataset use (1)

  • Biomni Third-party · Re-run by the benchmark authors rather than reported by the developer; the Chinese matrix marks it self-reported, which under this round's definition should be third-party and is recorded as an open inconsistency. †partly verified · evidence: link

Unconfirmed (6, not counted)

  • Co-Scientist Unconfirmed source · No report on this benchmark found in official materials. †disputed · evidence: link
  • GPT-Rosalind Unconfirmed source · No report found; the only evidence is a site page, so it is shown as unconfirmed. †disputed · evidence: no source link
  • Intern-S Unconfirmed source · No report on this benchmark found in official materials. †disputed · evidence: link
  • Quine Unconfirmed source · No report on this benchmark found in official materials. †disputed · evidence: link
  • SciReasoner Unconfirmed source · The paper text does not mention this benchmark; the only evidence is a site page. †disputed · evidence: no source link
  • VERID Unconfirmed source · No report found; the only evidence is a site page, so it is shown as unconfirmed. †disputed · evidence: no source link

Caveats

  1. Results mix agent design with LLM backbone; the authors do not separate them.
  2. Most of the 41 failed runs came from scaffolding problems, not model capability.
  3. Human leaderboard scores mix participant skill, effort and compute, so they are a rough reference only.
  4. Some authors are affiliated with the organization that maintains the evaluation package.
  5. Tasks are taken from public competitions, so training-data contamination is possible.

How to use it

Install the official package, prepare data with the per-task scripts, and run each task several times independently (four in the paper). Report both the penalized and unpenalized protocols, the LLM backbone, the budget and the time limit, and do not mix specialised agents with a bare backbone plus generic scaffolding in one table. Because tasks come from public competitions, assess training-data contamination.

Pairs well with: LAB-Bench / LABBench2 · BixBench · BioMysteryBench

Sources

  1. BioML-bench preprint, full text (preprint) ·
  2. bioRxiv version metadata (official site) ·
  3. Official repository (repository)
  4. Independent follow-up report (24 tasks) (third-party paper)
  5. Another follow-up benchmark, described by its authors as complementary (third-party paper)

Corrections

2 corrections
FieldWasNowBasisFound
ranking.headlinea headline percentile was shown for this benchmarkno headline number; the caveat text is displayed insteadsource2026-11-16
models_reported[model-biomni].relation_typeself-reported (Chinese matrix cell)benchmark-owner / third-party run, recorded as retrospective dataset use pending the matrix fixsource2026-11-16

Part of the AI for Bio benchmarks survey · all 44 benchmarks · open in the explorer. Record as of 2026-11-16.