AI for Bio · Benchmark 09

scIB (+Open Problems)

Default yardstick for single-cell integration, scoring batch removal against biological conservation; continued by Open Problems.

bench-09Type B+CEvidence L2Single-cell & perturbation● Active Board中文

Overview

NamescIB (+Open Problems) · aliases: scIB, scib, scib-metrics, Open Problems batch_integration, task_batch_integration
Full nameSingle-cell integration benchmark (scIB), with the Open Problems batch-integration task
Ambiguous names (not auto-linked)Open Problems
FamilyFamily: scIB metrics and package, the scib-metrics reimplementation, the Open Problems batch-integration task and the NeurIPS 2021 competition; scib and scib-metrics numbers differ.
  • scIB (paper and scib package) ·
  • scib-metrics (reimplementation) ·
  • NeurIPS 2021 multimodal single-cell competition · †
  • Open Problems batch_integration task · No sourced date†
DomainSingle-cell & perturbation
InclusionOriginal 30
TypeB C · Dataset suite and metrics toolkit plus a living platform
Evidence levelL2 · Retrospective, no blind test; author-annotation truth
PublisherHelmholtz Munich, Open Problems in Single-Cell Analysis (community consortium) (Mixed)
First release · paper v1
Basis: bioRxiv v1 of the scIB paper (Nature Methods online 2021-12-23; PyPI scib 1.0.0 2021-10-24) source
Other milestones
  • Preprint v1: · scIB paper bioRxiv v1 source
  • Journal online: · Nature Methods online (2022 issue) source
  • Code release: · PyPI scib 1.0.0 source
Latest update · new version
Basis: Release date of scib-metrics 0.6.2 on PyPI
scib-metrics 0.6.2 released on PyPI; scib GitHub release 1.2.1 on 2026-04-27 (rpy2 fix) after 1.2.0 added Moran's I.
Note: Site page also notes a task-repo v2.1.0-rc3 on 2026-11-03; not re-verified source
The most recent sourced event, not a claim that nothing changed since.
Next expectedLiving benchmark with no announced schedule; no component announces a next-version date.
Status● Active · Active via Open Problems living benchmark; integration task near saturation
ScaleOriginal paper: 16 methods, 13 tasks, 85 batches, 14 metrics; Open Problems: 12 tasks in the 2025 version.
  • Integration methods: 16 (Nature Methods 2022 (68 with preprocessing combinations))
  • Integration tasks: 13 (Nature Methods 2022)
  • Batches: 85 (Nature Methods 2022)
  • Metrics: 14 (Nature Methods 2022)
  • Cells: >1.2M cells (Nature Methods 2022)
  • Open Problems tasks: 12 (Nat Biotechnol 2025 version)
  • Batch-integration task datasets: 6 (Open Problems task page, read 2026-11)
Wet-lab refreshGrows from public sources · No new experiments; Open Problems adds datasets and tasks irregularly from public atlases.
Own wet lab needed to evaluate a new modelNo, run it as a download
Models reported1 counted (self 1, unconfirmed 2)
Unverified fields† family.members[2], family.members[3], ranking.entries, models_reported[1], models_reported[2]
As of2026-11-16

What it measures

  • Task: Single-cell data integration quality: batch removal against preservation of biological variation.
  • Truth: Author-provided cell-type annotations and batch metadata, not an experimental gold standard.
  • Primary metric: Overall = 0.4 × batch removal + 0.6 × bio-conservation
  • Secondary metrics: Batch: batch ASW, iLISI, kBET and others; Bio: ARI/NMI, cLISI, isolated label and others
  • Version traps: State scib or scib-metrics (numbers not comparable); Open Problems versions and dataset sets change over time.

Current ranking

Board

MetricOpen Problems batch integration: mean of 13 scaled scIB metrics per dataset, averaged over 6 datasets (failed or missing runs count as 0)
Eval setOpen Problems task_batch_integration, 6 CELLxGENE Census datasets; results run 2025-01 (website data commit 2025-02-10); page shows release 2.1.0-rc3 / stable v2.0.0
Snapshot stale
Boardhttps://openproblems.bio/benchmarks/batch_integration
Retrieved2026-11-16

Caveat. Ranks derived from the board's data file and include controls; results run 2025-01; AIDO.Cell and scPRINT-2 are not on the board; the 2022 paper has no foundation models.

No headline entry: The board orders by scaled overall mean but shows no numeric overall score; derived means are not used as a headline.

RankModelScore (verbatim)Metric · eval set / protocolSource type · dateSource and note
1embed_cell_types_jittered (positive control)No scoreOpen Problems batch integration: mean of 13 scaled scIB metrics per dataset, averaged over 6 datasets (failed or missing runs count as 0)
Open Problems task_batch_integration, 6 CELLxGENE Census datasets; results run 2025-01 (website data commit 2025-02-10); page shows release 2.1.0-rc3 / stable v2.0.0
Board
source Positive control, not a model; rank derived from the data file, page shows no overall score
4scanviNo scoreOpen Problems batch integration: mean of 13 scaled scIB metrics per dataset, averaged over 6 datasets (failed or missing runs count as 0)
Open Problems task_batch_integration, 6 CELLxGENE Census datasets; results run 2025-01 (website data commit 2025-02-10); page shows release 2.1.0-rc3 / stable v2.0.0
Board
source Not a site model; top non-control method; rank derived from the data file
18scgpt_zeroshotNo scoreOpen Problems batch integration: mean of 13 scaled scIB metrics per dataset, averaged over 6 datasets (failed or missing runs count as 0)
Open Problems task_batch_integration, 6 CELLxGENE Census datasets; results run 2025-01 (website data commit 2025-02-10); page shows release 2.1.0-rc3 / stable v2.0.0
Board
source Zero-shot scGPT; failed on 2 of 6 datasets, counted as 0; derived rank
23geneformerNo scoreOpen Problems batch integration: mean of 13 scaled scIB metrics per dataset, averaged over 6 datasets (failed or missing runs count as 0)
Open Problems task_batch_integration, 6 CELLxGENE Census datasets; results run 2025-01 (website data commit 2025-02-10); page shows release 2.1.0-rc3 / stable v2.0.0
Board
source Geneformer; 2 datasets missing (0), mean below the no-integration control; derived rank
25scprintNo scoreOpen Problems batch integration: mean of 13 scaled scIB metrics per dataset, averaged over 6 datasets (failed or missing runs count as 0)
Open Problems task_batch_integration, 6 CELLxGENE Census datasets; results run 2025-01 (website data commit 2025-02-10); page shows release 2.1.0-rc3 / stable v2.0.0
Board
source scPRINT v1 (not scPRINT-2); all results missing, so the rank reflects failure, not performance

Models reported

Grouped by relation type; each line shows its source type. Counted relations add up to 1. A pairing whose evidence rests only on this survey's own page is shown as unverified.

Developer report, official protocol (1)

Unconfirmed (2, not counted)

  • AIDO Cell Unconfirmed source · Model card claims zero-shot clustering SOTA; scIB not named, no numbers †disputed · evidence: link
  • scPRINT-2 Unconfirmed source · Abstract does not name scIB †disputed · evidence: link

Caveats

  1. Ground truth is author annotation, not experimental gold standard; metrics saturate, rankings are weight-sensitive, and scib and scib-metrics numbers are not comparable.
  2. A third-party study found zero-shot scGPT and Geneformer trail HVG, Harmony and scVI; baselines are fitted on target data and models are not fine-tuned.
  3. Batch ASW is near saturation; one study built embeddings with high scIB scores but very different structure; within-type variation is missed.
  4. Pretraining corpora may overlap evaluation data; there is no train/test split, integration and scoring use the same data.
  5. Integration scores map to no experimental hit rate; at most a proxy for annotation and atlas building.

How to use it

Install scib-metrics and call its Benchmarker with batch and cell-type keys, or submit a method through the Open Problems pipeline; no wet lab. State scib or scib-metrics, report weight sensitivity and within-type metrics; foundation models must add fitted HVG, Harmony and scVI baselines and separate zero-shot from fine-tuned. Pair with VCC perturbation.

Pairs well with: Virtual Cell Challenge and perturbation benchmarks · DREAM5 / BEELINE / CausalBench

Sources

  1. scIB paper (Nature Methods 2022) (journal)
  2. scIB preprint (bioRxiv 2020) (preprint)
  3. Open Problems batch-integration board (leaderboard)
  4. Open Problems board data file (repository)
  5. scib-metrics (PyPI) (package registry)
  6. scib (PyPI) (package registry)
  7. Open Problems platform preprint (preprint)
  8. Zero-shot foundation-model evaluation (Genome Biol 2025) (third-party paper)
  9. scIB-E extended metrics (Genome Biol 2025) (third-party paper)

Corrections

2 corrections
FieldWasNowBasisFound
latest_update / versionscib-metrics v0.5.8 latest; scib repository has no formal releasescib-metrics 0.6.2 (2026-11-01); scib GitHub releases 1.2.0 / 1.2.1 (2026-04)source2026-11-16
ranking.headlineheadline pointing to a derived overall meannullsource2026-11-16

Part of the AI for Bio benchmarks survey · all 44 benchmarks · open in the explorer. Record as of 2026-11-16.