Overview
| Name | BixBench · aliases: BixBench, BixBench v1.5, BixBench3 |
|---|---|
| Full name | A benchmark for bioinformatics analysis agents (BixBench) |
| Ambiguous names (not auto-linked) | Verified-50 |
| Family | Not a family entry. The successor is run by a different organisation with different tasks and metrics, so it is listed separately rather than as one family. |
| Domain | Text, LLMs & agents |
| Inclusion | Original 30 |
| Type | E · Executable containerised evaluation environment, graded on conclusions |
| Evidence level | L2 · Truth is the conclusions of published analyses, not experimental validation |
| Publisher | FutureHouse, ScienceMachine (Nonprofit institute) |
| First release | † partly verified · paper v1 Basis: The paper's arXiv v1 (2025-02-28); the dataset hosting page was created a day earlier, but creation does not prove public availability. Note: The hosting page's creation date is not used as first release source |
| Other milestones | |
| Latest update | † partly verified · board entry Basis: Entry dates on the successor's official leaderboard page A new model entry appeared on the successor's leaderboard; the original's latest events are a paper revision and a repository merge. Note: The board page carries per-entry publish dates but no overall snapshot date source The most recent sourced event, not a claim that nothing changed since. |
| Next expected | The organisers have not announced a new version or round for the original benchmark. |
| Status | ● Active · Active; the successor has an official board that keeps updating |
| Scale | First version: 53 capsules and 296 questions; the revision: about 61 capsules and 205 questions.
|
| Wet-lab refresh | Static data · No experiments; version updates are filtering revisions rather than new data. |
| Own wet lab needed to evaluate a new model | No, run it as a download |
| Models reported | 3 counted (board 1, self 1, modified protocol 1, unconfirmed 6) |
| Unverified fields | † first_release, latest_update, release_milestones.code_release, scale.items[2], models_reported[1], ranking.snapshot_date |
| As of | 2026-11-16 |
What it measures
- Task: Agents write and run code on real datasets to complete multi-step bioinformatics analyses, graded on their conclusions.
- Truth: Conclusions from expert analysis notebooks, not wet-lab measurements.
- Primary metric: Open-answer accuracy, graded by a model
- Secondary metrics: Multiple-choice accuracy, with and without a refusal option
- Version traps: State the version and subset: the first version, the revision, the third-party verified subset and the successor differ in question count and metric.
Current ranking
Board
| Metric | Average score over the successor's 20 tasks, scored programmatically against source papers |
|---|---|
| Eval set | BixBench3 (successor benchmark, 20 research-study-scale tasks) |
| Snapshot | |
| Board | https://advances.edisonscientific.com/benchmarks/bixbench3/ |
| Retrieved | 2026-11-16 |
Caveat. The only official board belongs to the successor, whose tasks and metric differ and do not compare with the original's accuracy; the original has no official board and no official verified subset, so those rows are self-reported or third-party numbers.
Shown in the overview table: Board
| Rank | Model | Score (verbatim) | Metric · eval set / protocol | Source type · date | Source and note |
|---|---|---|---|---|---|
| 1 | gemini-4-argon | 0.523 | Average score over the successor's 20 tasks, scored programmatically against source papers BixBench3 (successor benchmark, 20 research-study-scale tasks) | Board | source Board top entry; the entry's published date is 2026-10-16 |
| 2 | claude-fable-5.1 (with fallback) | 0.515 | Average score over the successor's 20 tasks, scored programmatically against source papers BixBench3 (successor benchmark, 20 research-study-scale tasks) | Board | source Model name copied verbatim from the board |
| 3 | claude-opus-5.5 | 0.502 | Average score over the successor's 20 tasks, scored programmatically against source papers BixBench3 (successor benchmark, 20 research-study-scale tasks) | Board | source |
| 4 | gpt-6-astra | 0.492 | Average score over the successor's 20 tasks, scored programmatically against source papers BixBench3 (successor benchmark, 20 research-study-scale tasks) | Board | source |
| 5 | grok-4.6 | 0.488 | Average score over the successor's 20 tasks, scored programmatically against source papers BixBench3 (successor benchmark, 20 research-study-scale tasks) | Board | source The board lists 20 models, with the last at 0.000 |
| — | Biomni | 52.2% (full set) / 88.7% (Verified-50) | Average score over the successor's 20 tasks, scored programmatically against source papers Original BixBench: full set, plus a third-party verified subset of 50 questions | Self-reported | source On the original, not the successor; run by its own developer, so not ranked against the board |
| — | Kosmos/Edison Analysis | 46% (Edison self-report, full set); 42.4% full set / 78.0% Verified-50 (Phylo re-run) | Average score over the successor's 20 tasks, scored programmatically against source papers Original BixBench: full set, plus a third-party verified subset of 50 questions | Third-party | source On the original; the self-reporting party is tied to the successor's operator |
Models reported
Grouped by relation type; each line shows its source type. Counted relations add up to 3. A pairing whose evidence rests only on this survey's own page is shown as unverified.
Official leaderboard or results (1)
- Frontier general-purpose LLMs Board BixBench3 · Successor's official board; counted as one family cell · evidence: link
Developer report, official protocol (1)
- Kosmos / Edison Analysis Self-reported original BixBench · Self-report on the original; the product's mapping to this site entry is unclear and it is not on the successor's board low confidence †unverified · evidence: link
Developer report, modified protocol (1)
- Biomni Self-reported BixBench full + verified-50 · Run by its developer on the full set and a verified subset with its own agent harness †partly verified · evidence: link
Unconfirmed (6, not counted)
- Co-Scientist Unconfirmed source · No report found in official materials †disputed · evidence: link
- GPT-Rosalind Unconfirmed source · Version unspecified and the numbers appear only in second-hand reports †disputed · evidence: link
- Intern-S Unconfirmed source · No report found in official materials †disputed · evidence: link
- Quine Unconfirmed source · No report found in official materials †disputed · evidence: link
- SciReasoner Unconfirmed source · Not found in the paper text †disputed · evidence: no source link
- VERID Unconfirmed source · No report found in official materials †disputed · evidence: no source link
Caveats
- The truth comes from a single analyst, with ambiguous questions and some wrong answers; scores on a third-party verified subset are markedly higher.
- The multiple-choice version carries little signal, and open answers are graded by a model, which is itself a source of error.
- The data and questions are public, so contamination is a risk and there is no time freeze.
- Scores must state the version: question counts and subsets differ, so numbers do not compare directly.
- The successor is operated by a commercial organisation that also self-reports results, which is a conflict of interest.
How to use it
Run the agent inside the official container environment, produce conclusions per capsule and score them with the official grader; no wet lab of your own is needed. Report the version, question count, whether a verified subset was used, the grading model and the agent harness, and separate open answers from multiple choice. Successor scores must not be mixed with the original's. Pair with LAB-Bench and BioML-bench.
Pairs well with: LAB-Bench / LABBench2 · BioML-bench · BioMysteryBench · BioASQ
Sources
- Original paper (preprint)
- Official repository and evaluation environment (repository)
- Official dataset hosting page (repository)
- Official board page of the successor (leaderboard)
- Blog describing a third-party verified subset and re-runs (company blog)
Corrections
2 corrections
Part of the AI for Bio benchmarks survey · all 44 benchmarks · open in the explorer. Record as of 2026-11-16.