AI for Bio · Benchmark 29

BixBench

Agents write and run code on real data in a container to do bioinformatics analyses, graded on their conclusions.

bench-29Type EEvidence L2Text, LLMs & agents● Active Board中文

Overview

NameBixBench · aliases: BixBench, BixBench v1.5, BixBench3
Full nameA benchmark for bioinformatics analysis agents (BixBench)
Ambiguous names (not auto-linked)Verified-50
FamilyNot a family entry. The successor is run by a different organisation with different tasks and metrics, so it is listed separately rather than as one family.
DomainText, LLMs & agents
InclusionOriginal 30
TypeE · Executable containerised evaluation environment, graded on conclusions
Evidence levelL2 · Truth is the conclusions of published analyses, not experimental validation
PublisherFutureHouse, ScienceMachine (Nonprofit institute)
First release† partly verified · paper v1
Basis: The paper's arXiv v1 (2025-02-28); the dataset hosting page was created a day earlier, but creation does not prove public availability.
Note: The hosting page's creation date is not used as first release source
Other milestones
  • Preprint v1: † · The paper's arXiv v1 source
  • Code release: † · Last merge in the official repository source
Latest update† partly verified · board entry
Basis: Entry dates on the successor's official leaderboard page
A new model entry appeared on the successor's leaderboard; the original's latest events are a paper revision and a repository merge.
Note: The board page carries per-entry publish dates but no overall snapshot date source
The most recent sourced event, not a claim that nothing changed since.
Next expectedThe organisers have not announced a new version or round for the original benchmark.
Status● Active · Active; the successor has an official board that keeps updating
ScaleFirst version: 53 capsules and 296 questions; the revision: about 61 capsules and 205 questions.
  • Capsules: 53 (v1)
  • Questions: 296 (v1)
  • Capsules: about 61 (v3 / repository v1.5)
  • Questions: 205 (v3 / repository v1.5)
  • Tasks in the successor: 20 (successor)
Wet-lab refreshStatic data · No experiments; version updates are filtering revisions rather than new data.
Own wet lab needed to evaluate a new modelNo, run it as a download
Models reported3 counted (board 1, self 1, modified protocol 1, unconfirmed 6)
Unverified fields† first_release, latest_update, release_milestones.code_release, scale.items[2], models_reported[1], ranking.snapshot_date
As of2026-11-16

What it measures

  • Task: Agents write and run code on real datasets to complete multi-step bioinformatics analyses, graded on their conclusions.
  • Truth: Conclusions from expert analysis notebooks, not wet-lab measurements.
  • Primary metric: Open-answer accuracy, graded by a model
  • Secondary metrics: Multiple-choice accuracy, with and without a refusal option
  • Version traps: State the version and subset: the first version, the revision, the third-party verified subset and the successor differ in question count and metric.

Current ranking

Board

MetricAverage score over the successor's 20 tasks, scored programmatically against source papers
Eval setBixBench3 (successor benchmark, 20 research-study-scale tasks)
Snapshot
Boardhttps://advances.edisonscientific.com/benchmarks/bixbench3/
Retrieved2026-11-16

Caveat. The only official board belongs to the successor, whose tasks and metric differ and do not compare with the original's accuracy; the original has no official board and no official verified subset, so those rows are self-reported or third-party numbers.

Shown in the overview table:

0.523
Average score over the successor's 20 tasks, scored programmatically against source papers · BixBench3 (successor benchmark, 20 research-study-scale tasks)
Board ·

RankModelScore (verbatim)Metric · eval set / protocolSource type · dateSource and note
1gemini-4-argon0.523Average score over the successor's 20 tasks, scored programmatically against source papers
BixBench3 (successor benchmark, 20 research-study-scale tasks)
Board
source Board top entry; the entry's published date is 2026-10-16
2claude-fable-5.1 (with fallback)0.515Average score over the successor's 20 tasks, scored programmatically against source papers
BixBench3 (successor benchmark, 20 research-study-scale tasks)
Board
source Model name copied verbatim from the board
3claude-opus-5.50.502Average score over the successor's 20 tasks, scored programmatically against source papers
BixBench3 (successor benchmark, 20 research-study-scale tasks)
Board
source
4gpt-6-astra0.492Average score over the successor's 20 tasks, scored programmatically against source papers
BixBench3 (successor benchmark, 20 research-study-scale tasks)
Board
source
5grok-4.60.488Average score over the successor's 20 tasks, scored programmatically against source papers
BixBench3 (successor benchmark, 20 research-study-scale tasks)
Board
source The board lists 20 models, with the last at 0.000
—Biomni52.2% (full set) / 88.7% (Verified-50)Average score over the successor's 20 tasks, scored programmatically against source papers
Original BixBench: full set, plus a third-party verified subset of 50 questions
Self-reported
source On the original, not the successor; run by its own developer, so not ranked against the board
—Kosmos/Edison Analysis46% (Edison self-report, full set); 42.4% full set / 78.0% Verified-50 (Phylo re-run)Average score over the successor's 20 tasks, scored programmatically against source papers
Original BixBench: full set, plus a third-party verified subset of 50 questions
Third-party
source On the original; the self-reporting party is tied to the successor's operator

Models reported

Grouped by relation type; each line shows its source type. Counted relations add up to 3. A pairing whose evidence rests only on this survey's own page is shown as unverified.

Official leaderboard or results (1)

Developer report, official protocol (1)

  • Kosmos / Edison Analysis Self-reported original BixBench · Self-report on the original; the product's mapping to this site entry is unclear and it is not on the successor's board low confidence †unverified · evidence: link

Developer report, modified protocol (1)

  • Biomni Self-reported BixBench full + verified-50 · Run by its developer on the full set and a verified subset with its own agent harness †partly verified · evidence: link

Unconfirmed (6, not counted)

  • Co-Scientist Unconfirmed source · No report found in official materials †disputed · evidence: link
  • GPT-Rosalind Unconfirmed source · Version unspecified and the numbers appear only in second-hand reports †disputed · evidence: link
  • Intern-S Unconfirmed source · No report found in official materials †disputed · evidence: link
  • Quine Unconfirmed source · No report found in official materials †disputed · evidence: link
  • SciReasoner Unconfirmed source · Not found in the paper text †disputed · evidence: no source link
  • VERID Unconfirmed source · No report found in official materials †disputed · evidence: no source link

Caveats

  1. The truth comes from a single analyst, with ambiguous questions and some wrong answers; scores on a third-party verified subset are markedly higher.
  2. The multiple-choice version carries little signal, and open answers are graded by a model, which is itself a source of error.
  3. The data and questions are public, so contamination is a risk and there is no time freeze.
  4. Scores must state the version: question counts and subsets differ, so numbers do not compare directly.
  5. The successor is operated by a commercial organisation that also self-reports results, which is a conflict of interest.

How to use it

Run the agent inside the official container environment, produce conclusions per capsule and score them with the official grader; no wet lab of your own is needed. Report the version, question count, whether a verified subset was used, the grading model and the agent harness, and separate open answers from multiple choice. Successor scores must not be mixed with the original's. Pair with LAB-Bench and BioML-bench.

Pairs well with: LAB-Bench / LABBench2 · BioML-bench · BioMysteryBench · BioASQ

Sources

  1. Original paper (preprint)
  2. Official repository and evaluation environment (repository)
  3. Official dataset hosting page (repository)
  4. Official board page of the successor (leaderboard)
  5. Blog describing a third-party verified subset and re-runs (company blog)

Corrections

2 corrections
FieldWasNowBasisFound
ranking.eval_setall ranking rows labelled as the successor's task setrows on the original benchmark are labelled as the original's full set plus a third-party verified subsetsource2026-11-16
first_release2025-032025-02-28 (paper arXiv v1)source2026-11-16

Part of the AI for Bio benchmarks survey · all 44 benchmarks · open in the explorer. Record as of 2026-11-16.