AI for Bio · Benchmark 24

Virtual Cell Challenge and perturbation benchmarks

The only challenge generating Perturb-seq data for evaluation, yet linear and mean baselines remain hard to beat

bench-24Type A+B+CEvidence L2–L4Single-cell & perturbation● Active Board中文

Overview

NameVirtual Cell Challenge and perturbation benchmarks · aliases: Virtual Cell Challenge, VCC, VCC 2025, VCC 2026, PerturBench, cell-eval
Full nameThe Virtual Cell Challenge plus PerturBench and related single-cell perturbation benchmarks
Ambiguous names (not auto-linked)Norman, Replogle, Adamson
FamilyVCC is a yearly blind challenge while PerturBench and the Norman, Replogle and Adamson suites are static benchmarks and cell-eval is only a tool; the first release is the earliest member, PerturBench, not the VCC launch.
  • PerturBench ·
  • Virtual Cell Challenge 2025 ·
  • Virtual Cell Challenge 2026 ·
DomainSingle-cell & perturbation (also: Genomics & RNA)
InclusionOriginal 30
TypeA B C · A blind challenge plus static perturbation suites and an evaluation tool
Evidence levelL2–L4 · VCC uses a hidden test set; the static suites and baseline critiques are L2-L3
PublisherArc Institute, Altos Labs (Nonprofit institute)
First release · paper v1
Basis: Earliest family member: PerturBench's arXiv v1; VCC itself launched on 2025-06-26 source
Other milestones
  • Preprint v1: · PerturBench arXiv 2408.10609 v1 source
  • Challenge opens: · VCC 2025 launch with data and leaderboard opened source
  • Official announcement: · VCC 2026 announcement with registration open source
Latest update† partly verified · new round
Basis: Release of the VCC 2026 final test set, per the organizer's published schedule
The VCC 2026 final test set was released and final submissions closed on 2026-11-05; as of 2026-11-16 no results have been published.
Note: The date could not be matched to a dated official passage, so it is shown as unverified. source
The most recent sourced event, not a claim that nothing changed since.
Next expected† unverified · results
Basis: The organizer schedule says the 2026 results come in mid-to-late November, while a press release says December source
Status● Active · Active: VCC 2026 has closed and its results are not yet published
ScaleVCC 2025: about 300,000 cells, 300 perturbations and 300+ teams in the final round; VCC 2026: six cell lines with cell counts unpublished.
  • Cells, H1 hESC: ~300000 cells (VCC 2025)
  • CRISPRi perturbations: 300 (VCC 2025)
  • Registrations: 5000+ people (VCC 2025)
  • Teams in the final round: 300+ (VCC 2025)
  • Cell lines, 3 validation and 3 final, not re-confirmed: 6 (VCC 2026)
  • Cells: (VCC 2026)
Wet-lab refreshFresh experiments each round · Yes, yearly: the organizers run their own Perturb-seq; the 2025 edition had about 300,000 cells and the 2026 cell count is unpublished.
Own wet lab needed to evaluate a new modelNo, run it as a download
Models reported9 counted (board 1, modified protocol 6, retrospective 2)
Unverified fields† latest_update, next_expected, scale.items[4], scale.items[5], ranking.entries[4], family.members[0]
As of2026-11-16

What it measures

  • Task: Predict how a cell's transcriptome changes when a gene is knocked down, including unseen perturbations and unseen cell backgrounds, with the 2026 edition zero-shot.
  • Truth: Newly measured CRISPRi Perturb-seq data from the organizers; the static suites use published Norman, Replogle and Adamson data.
  • Primary metric: 2025: aggregate rank over PDS, DES and MAE
  • Secondary metrics: Generalist Prize: mean rank across 7 metrics; 2026: a six-metric aggregate in the new cell-eval; Pearson delta and AUPRC
  • Version traps: The 2025 data are now public and no longer blind, the two editions use different metric sets, and the L1/L2 forms of PDS are sensitive to prediction scale, so ranks do not compare across editions.

Current ranking

Board

MetricVCC 2025 final rank aggregating PDS, DES and MAE, with the Generalist Prize as mean rank over 7 metrics
Eval setVirtual Cell Challenge 2025 final test set (H1 hESC, held-out CRISPRi perturbations)
Snapshot
Boardhttps://arcinstitute.org/news/virtual-cell-challenge-2025-wrap-up
Retrieved2026-11-16

Caveat. The 2026 edition closed on 2026-11-05 but had announced no standing as of 2026-11-16, so the current standing is still 2025; that board publishes ranks only, never scores.

Shown in the overview table:

rank 1 of 300+ final-round teams (board publishes ranks only, no numeric scores)
VCC 2025 final rank aggregating PDS, DES and MAE, with the Generalist Prize as mean rank over 7 metrics · Virtual Cell Challenge 2025 final test set (H1 hESC, held-out CRISPRi perturbations)
Board ·

RankModelScore (verbatim)Metric · eval set / protocolSource type · dateSource and note
1Team BM_xTVC, xTrimoSCPerturb (improved scFoundation architecture with protein-model embeddings)rank 1 of 300+ final-round teams (board publishes ranks only, no numeric scores)VCC 2025 final rank aggregating PDS, DES and MAE, with the Generalist Prize as mean rank over 7 metrics
Virtual Cell Challenge 2025 final test set (H1 hESC, held-out CRISPRi perturbations)
Board
source Overall winner; the organizer wrap-up publishes ranks and prizes only, with no numeric score for any team.
2Team XLearning Lab (fully connected network, ESM-2 embeddings, residual delta prediction)No scoreVCC 2025 final rank aggregating PDS, DES and MAE, with the Generalist Prize as mean rank over 7 metrics
Virtual Cell Challenge 2025 final test set (H1 hESC, held-out CRISPRi perturbations)
Board
source Not a site model; no numeric score published.
3Team Outlier, TransPert (summary-level statistical framework)No scoreVCC 2025 final rank aggregating PDS, DES and MAE, with the Generalist Prize as mean rank over 7 metrics
Virtual Cell Challenge 2025 final test set (H1 hESC, held-out CRISPRi perturbations)
Board
source Not a site model; no numeric score published.
—Team Altos Labs, go-with-the-flow (flow matching with a custom U-Net), Generalist PrizeNo scoreVCC 2025 final rank aggregating PDS, DES and MAE, with the Generalist Prize as mean rank over 7 metrics
Virtual Cell Challenge 2025 final test set (H1 hESC, held-out CRISPRi perturbations)
Board
source Not a site model; the prize goes by mean rank over 7 metrics among the top 50 final entries, with no numbers.
—STATEWithheld: A different protocol and not a board result, so nothing is shown in the ranking row.VCC 2025 final rank aggregating PDS, DES and MAE, with the Generalist Prize as mean rank over 7 metrics
Virtual Cell Challenge 2025 final test set (H1 hESC, held-out CRISPRi perturbations)
Self-reported
source Not an entry: the organizers used four metrics from its paper to define the Generalist Prize, and its self-reported Tahoe-100M results use a different protocol and are not board results.

Models reported

Grouped by relation type; each line shows its source type. Counted relations add up to 9. A pairing whose evidence rests only on this survey's own page is shown as unverified.

Official leaderboard or results (1)

Developer report, modified protocol (6)

  • C2S-Scale Self-reported · Uses its own datasets and metrics, not the VCC protocol. · evidence: link
  • CellFM Self-reported · Adamson and Norman data through GEARS, not VCC. · evidence: link
  • Stack Self-reported · Own perturbation sets; its VCC claim is rejected on the model page. · evidence: link
  • STATE Self-reported · Self-reported with cell-eval on Tahoe-100M; its VCC blind score is unpublished. · evidence: link
  • Tahoe-x1 Self-reported · An own benchmark, explicitly not VCC. · evidence: link
  • X-Cell Self-reported · Own held-out iPSC set; VCC participation unconfirmed. · evidence: link

Retrospective dataset use (2)

  • Lingshu-Cell Third-party · A post-hoc evaluation on the VCC H1 leaderboard rather than participation. · evidence: link
  • scGPT / Geneformer Third-party · Independent third-party evaluations on Norman and Adamson data. · evidence: link

Caveats

  1. The 2025 metrics can be gamed by scale and transformations: a third-party analysis shows L1/L2 PDS is sensitive to prediction scale and almost all submissions scored worse than baseline on MAE.
  2. Several independent evaluations find deep and foundation models do not reliably beat mean, no-change or linear baselines on perturbation prediction.
  3. The board gives ranks without numbers, and with 300 perturbations the organizers never state whether rank gaps exceed noise.
  4. The 2025 edition used a single cell line so cross-cell generalization was untested, and the full data are now public and no longer blind.
  5. External pretraining and public Perturb-seq may overlap the test targets, and there is no prospective wet-lab validation nor evidence linking scores to target-discovery hit rates.

How to use it

The 2025 data are public and can be run statically, but a real blind test means entering the next edition. Always report mean, no-change and linear baselines, use several metrics such as cosine PDS, Pearson delta and AUPRC plus PerturBench ranks and RMSD, and never rely on a single PDS. Generalization to one's own cell lines needs new Perturb-seq, whose cost is undisclosed.

Pairs well with: DREAM5 / BEELINE / CausalBench · scIB (+Open Problems) · Basenji/Enformer test set

Sources

  1. Organizer wrap-up page for VCC 2025 (official site) ·
  2. VCC 2026 announcement (official site) ·
  3. PerturBench paper (preprint) ·
  4. Third-party comparison of deep models against linear baselines (third-party paper)
  5. Third-party analysis of the PDS metric (third-party paper)
  6. PertEval-scFM (paper)

Corrections

4 corrections
FieldWasNowBasisFound
first_release2025 (VCC launch treated as the family's first release)2024-08-20, PerturBench arXiv v1, with the VCC 2025 launch (2025-06-26) kept as a membersource2026-11-16
scale2026 edition: six cell lines, zero-shot, cell counts implied2025 figures verified; the 2026 cell count is unpublished and 'six cell lines' was not re-confirmedsource2026-11-16
latest_updateVCC 2026 in progress, without a dated event2026-10-22 final test set released; submissions closed 2026-11-05; winners unannounced as of 2026-11-16 (flagged unverified)source2026-11-16
models_reportedSTATE, Stack, X-Cell and others listed as VCC resultsself-reported-modified-protocol: own datasets and metrics, not the VCC boardsource2026-11-16

Part of the AI for Bio benchmarks survey · all 44 benchmarks · open in the explorer. Record as of 2026-11-16.