Overview
| Name | BioML-bench · aliases: BioML-bench, BioML-Bench, biomlbench |
|---|---|
| Full name | The name is the full name; sources give no expanded form. |
| Ambiguous names (not auto-linked) | None |
| Family | Not a family entry. One task set with no family members; BioXArena is a separate follow-up benchmark. |
| Domain | Text, LLMs & agents |
| Inclusion | Supplementary entry |
| Type | E · End-to-end agent task environment |
| Evidence level | L2 · Truth hidden from the agent, but data come from already public competitions |
| Publisher | ScienceMachine, University of Cambridge, Shift Bioscience, University of Toronto, Vector Institute (Mixed) |
| First release | · paper v1 Basis: bioRxiv preprint v1 (the 2025-05-19 first repository commit predates the paper and is a note only) source |
| Other milestones | |
| Latest update | · paper revision Basis: bioRxiv version metadata Preprint v3; last repository commit 2025-09-14 (documentation fixes), maintenance only since. source The most recent sourced event, not a claim that nothing changed since. |
| Next expected | The maintainers have announced no next version or new round. |
| Status | ◐ Low-frequency · Low frequency: no new version after preprint v3; independent follow-up use in 2026. |
| Scale | Four domains; tasks taken from public competitions and datasets, with a follow-up report citing 24 tasks.
|
| Wet-lab refresh | Static data · Produces no new experimental data; tasks and truth come from existing public competitions and datasets. |
| Own wet lab needed to evaluate a new model | No, run it as a download |
| Models reported | 1 counted (retrospective 1, unconfirmed 6) |
| Unverified fields | † ranking.entries[0].date, scale.items[2], models_reported[model-gpt-rosalind], models_reported[model-scireasoner], models_reported[model-verid], models_reported[model-co-scientist], models_reported[model-intern-s], models_reported[model-quine] |
| As of | 2026-11-16 |
What it measures
- Task: Given a task description, training data and a submission example, the agent builds a model and submits predictions.
- Truth: Hidden test labels from public competitions and datasets; human baselines come from public leaderboards.
- Primary metric: Percentile against human leaderboards, with failed runs penalized
- Secondary metrics: Each task's native metric (AUROC, Spearman and others); Mean rank; Number of failed runs
- Version traps: State whether failure penalties are applied, which task-set version, which LLM backbone, and the budget and time limits.
Current ranking
Self-reported
| Metric | Mean leaderboard percentile against human public-leaderboard baselines, failed runs penalized |
|---|---|
| Eval set | BioML-bench tasks across 4 domains (24 tasks in the follow-up report), 4 runs per agent per task |
| Snapshot | |
| Board | No official board |
| Retrieved | 2026-11-16 |
Caveat. No official live leaderboard; numbers come from different papers and protocols and are not ranked against each other. Several agent models have no sourced number here.
No headline entry: The source has no rank-1 entry: entries are unranked or protocol-specific, so the caveat is shown as for a not-comparable cell.
| Rank | Model | Score (verbatim) | Metric · eval set / protocol | Source type · date | Source and note |
|---|---|---|---|---|---|
| — | Biomni | 34.45 ± 5.36 (leaderboard percentile) | Mean leaderboard percentile against human public-leaderboard baselines, failed runs penalized BioML-bench tasks across 4 domains (24 tasks in the follow-up report), 4 runs per agent per task | Self-reported † | source Best of the four agents re-run by the benchmark authors; mean rank 1.88 +/- 0.18, no failed tasks. |
| — | AutoScientists | 74.4% (mean leaderboard percentile, 24 tasks) | Mean leaderboard percentile against human public-leaderboard baselines, failed runs penalized BioML-bench tasks across 4 domains (24 tasks in the follow-up report), 4 runs per agent per task | Self-reported | source Not a site model; developer-reported with partly different hardware and protocol, and the report does not say whether the protocol matches the row above. |
Models reported
Grouped by relation type; each line shows its source type. Counted relations add up to 1. A pairing whose evidence rests only on this survey's own page is shown as unverified.
Retrospective dataset use (1)
- Biomni Third-party · Re-run by the benchmark authors rather than reported by the developer; the Chinese matrix marks it self-reported, which under this round's definition should be third-party and is recorded as an open inconsistency. †partly verified · evidence: link
Unconfirmed (6, not counted)
- Co-Scientist Unconfirmed source · No report on this benchmark found in official materials. †disputed · evidence: link
- GPT-Rosalind Unconfirmed source · No report found; the only evidence is a site page, so it is shown as unconfirmed. †disputed · evidence: no source link
- Intern-S Unconfirmed source · No report on this benchmark found in official materials. †disputed · evidence: link
- Quine Unconfirmed source · No report on this benchmark found in official materials. †disputed · evidence: link
- SciReasoner Unconfirmed source · The paper text does not mention this benchmark; the only evidence is a site page. †disputed · evidence: no source link
- VERID Unconfirmed source · No report found; the only evidence is a site page, so it is shown as unconfirmed. †disputed · evidence: no source link
Caveats
- Results mix agent design with LLM backbone; the authors do not separate them.
- Most of the 41 failed runs came from scaffolding problems, not model capability.
- Human leaderboard scores mix participant skill, effort and compute, so they are a rough reference only.
- Some authors are affiliated with the organization that maintains the evaluation package.
- Tasks are taken from public competitions, so training-data contamination is possible.
How to use it
Install the official package, prepare data with the per-task scripts, and run each task several times independently (four in the paper). Report both the penalized and unpenalized protocols, the LLM backbone, the budget and the time limit, and do not mix specialised agents with a bare backbone plus generic scaffolding in one table. Because tasks come from public competitions, assess training-data contamination.
Pairs well with: LAB-Bench / LABBench2 · BixBench · BioMysteryBench
Sources
- BioML-bench preprint, full text (preprint) ·
- bioRxiv version metadata (official site) ·
- Official repository (repository)
- Independent follow-up report (24 tasks) (third-party paper)
- Another follow-up benchmark, described by its authors as complementary (third-party paper)
Corrections
2 corrections
| Field | Was | Now | Basis | Found |
|---|---|---|---|---|
| ranking.headline | a headline percentile was shown for this benchmark | no headline number; the caveat text is displayed instead | source | 2026-11-16 |
| models_reported[model-biomni].relation_type | self-reported (Chinese matrix cell) | benchmark-owner / third-party run, recorded as retrospective dataset use pending the matrix fix | source | 2026-11-16 |
Part of the AI for Bio benchmarks survey · all 44 benchmarks · open in the explorer. Record as of 2026-11-16.