Overview
| Name | scIB (+Open Problems) · aliases: scIB, scib, scib-metrics, Open Problems batch_integration, task_batch_integration |
|---|---|
| Full name | Single-cell integration benchmark (scIB), with the Open Problems batch-integration task |
| Ambiguous names (not auto-linked) | Open Problems |
| Family | Family: scIB metrics and package, the scib-metrics reimplementation, the Open Problems batch-integration task and the NeurIPS 2021 competition; scib and scib-metrics numbers differ.
|
| Domain | Single-cell & perturbation |
| Inclusion | Original 30 |
| Type | B C · Dataset suite and metrics toolkit plus a living platform |
| Evidence level | L2 · Retrospective, no blind test; author-annotation truth |
| Publisher | Helmholtz Munich, Open Problems in Single-Cell Analysis (community consortium) (Mixed) |
| First release | · paper v1 Basis: bioRxiv v1 of the scIB paper (Nature Methods online 2021-12-23; PyPI scib 1.0.0 2021-10-24) source |
| Other milestones | |
| Latest update | · new version Basis: Release date of scib-metrics 0.6.2 on PyPI scib-metrics 0.6.2 released on PyPI; scib GitHub release 1.2.1 on 2026-04-27 (rpy2 fix) after 1.2.0 added Moran's I. Note: Site page also notes a task-repo v2.1.0-rc3 on 2026-11-03; not re-verified source The most recent sourced event, not a claim that nothing changed since. |
| Next expected | Living benchmark with no announced schedule; no component announces a next-version date. |
| Status | ● Active · Active via Open Problems living benchmark; integration task near saturation |
| Scale | Original paper: 16 methods, 13 tasks, 85 batches, 14 metrics; Open Problems: 12 tasks in the 2025 version.
|
| Wet-lab refresh | Grows from public sources · No new experiments; Open Problems adds datasets and tasks irregularly from public atlases. |
| Own wet lab needed to evaluate a new model | No, run it as a download |
| Models reported | 1 counted (self 1, unconfirmed 2) |
| Unverified fields | † family.members[2], family.members[3], ranking.entries, models_reported[1], models_reported[2] |
| As of | 2026-11-16 |
What it measures
- Task: Single-cell data integration quality: batch removal against preservation of biological variation.
- Truth: Author-provided cell-type annotations and batch metadata, not an experimental gold standard.
- Primary metric: Overall = 0.4 × batch removal + 0.6 × bio-conservation
- Secondary metrics: Batch: batch ASW, iLISI, kBET and others; Bio: ARI/NMI, cLISI, isolated label and others
- Version traps: State scib or scib-metrics (numbers not comparable); Open Problems versions and dataset sets change over time.
Current ranking
Board
| Metric | Open Problems batch integration: mean of 13 scaled scIB metrics per dataset, averaged over 6 datasets (failed or missing runs count as 0) |
|---|---|
| Eval set | Open Problems task_batch_integration, 6 CELLxGENE Census datasets; results run 2025-01 (website data commit 2025-02-10); page shows release 2.1.0-rc3 / stable v2.0.0 |
| Snapshot | stale |
| Board | https://openproblems.bio/benchmarks/batch_integration |
| Retrieved | 2026-11-16 |
Caveat. Ranks derived from the board's data file and include controls; results run 2025-01; AIDO.Cell and scPRINT-2 are not on the board; the 2022 paper has no foundation models.
No headline entry: The board orders by scaled overall mean but shows no numeric overall score; derived means are not used as a headline.
| Rank | Model | Score (verbatim) | Metric · eval set / protocol | Source type · date | Source and note |
|---|---|---|---|---|---|
| 1 | embed_cell_types_jittered (positive control) | No score | Open Problems batch integration: mean of 13 scaled scIB metrics per dataset, averaged over 6 datasets (failed or missing runs count as 0) Open Problems task_batch_integration, 6 CELLxGENE Census datasets; results run 2025-01 (website data commit 2025-02-10); page shows release 2.1.0-rc3 / stable v2.0.0 | Board | source Positive control, not a model; rank derived from the data file, page shows no overall score |
| 4 | scanvi | No score | Open Problems batch integration: mean of 13 scaled scIB metrics per dataset, averaged over 6 datasets (failed or missing runs count as 0) Open Problems task_batch_integration, 6 CELLxGENE Census datasets; results run 2025-01 (website data commit 2025-02-10); page shows release 2.1.0-rc3 / stable v2.0.0 | Board | source Not a site model; top non-control method; rank derived from the data file |
| 18 | scgpt_zeroshot | No score | Open Problems batch integration: mean of 13 scaled scIB metrics per dataset, averaged over 6 datasets (failed or missing runs count as 0) Open Problems task_batch_integration, 6 CELLxGENE Census datasets; results run 2025-01 (website data commit 2025-02-10); page shows release 2.1.0-rc3 / stable v2.0.0 | Board | source Zero-shot scGPT; failed on 2 of 6 datasets, counted as 0; derived rank |
| 23 | geneformer | No score | Open Problems batch integration: mean of 13 scaled scIB metrics per dataset, averaged over 6 datasets (failed or missing runs count as 0) Open Problems task_batch_integration, 6 CELLxGENE Census datasets; results run 2025-01 (website data commit 2025-02-10); page shows release 2.1.0-rc3 / stable v2.0.0 | Board | source Geneformer; 2 datasets missing (0), mean below the no-integration control; derived rank |
| 25 | scprint | No score | Open Problems batch integration: mean of 13 scaled scIB metrics per dataset, averaged over 6 datasets (failed or missing runs count as 0) Open Problems task_batch_integration, 6 CELLxGENE Census datasets; results run 2025-01 (website data commit 2025-02-10); page shows release 2.1.0-rc3 / stable v2.0.0 | Board | source scPRINT v1 (not scPRINT-2); all results missing, so the rank reflects failure, not performance |
Models reported
Grouped by relation type; each line shows its source type. Counted relations add up to 1. A pairing whose evidence rests only on this survey's own page is shown as unverified.
Developer report, official protocol (1)
- scGPT / Geneformer Self-reported PBMC 10K · scGPT only (PBMC10K), self-reported; Geneformer not applicable; family entry †partly verified · evidence: link
Unconfirmed (2, not counted)
- AIDO Cell Unconfirmed source · Model card claims zero-shot clustering SOTA; scIB not named, no numbers †disputed · evidence: link
- scPRINT-2 Unconfirmed source · Abstract does not name scIB †disputed · evidence: link
Caveats
- Ground truth is author annotation, not experimental gold standard; metrics saturate, rankings are weight-sensitive, and scib and scib-metrics numbers are not comparable.
- A third-party study found zero-shot scGPT and Geneformer trail HVG, Harmony and scVI; baselines are fitted on target data and models are not fine-tuned.
- Batch ASW is near saturation; one study built embeddings with high scIB scores but very different structure; within-type variation is missed.
- Pretraining corpora may overlap evaluation data; there is no train/test split, integration and scoring use the same data.
- Integration scores map to no experimental hit rate; at most a proxy for annotation and atlas building.
How to use it
Install scib-metrics and call its Benchmarker with batch and cell-type keys, or submit a method through the Open Problems pipeline; no wet lab. State scib or scib-metrics, report weight sensitivity and within-type metrics; foundation models must add fitted HVG, Harmony and scVI baselines and separate zero-shot from fine-tuned. Pair with VCC perturbation.
Pairs well with: Virtual Cell Challenge and perturbation benchmarks · DREAM5 / BEELINE / CausalBench
Sources
- scIB paper (Nature Methods 2022) (journal)
- scIB preprint (bioRxiv 2020) (preprint)
- Open Problems batch-integration board (leaderboard)
- Open Problems board data file (repository)
- scib-metrics (PyPI) (package registry)
- scib (PyPI) (package registry)
- Open Problems platform preprint (preprint)
- Zero-shot foundation-model evaluation (Genome Biol 2025) (third-party paper)
- scIB-E extended metrics (Genome Biol 2025) (third-party paper)
Corrections
2 corrections
Part of the AI for Bio benchmarks survey · all 44 benchmarks · open in the explorer. Record as of 2026-11-16.