AI for Bio · Benchmark 21

DREAM5 / BEELINE / CausalBench

Three generations of GRN inference evaluation whose scores mainly reflect gold-standard quality

bench-21Type A+B+CEvidence L2–L5Genomics & RNA◐ Low-frequency Not comparable中文

Overview

NameDREAM5 / BEELINE / CausalBench · aliases: DREAM5, DREAM5 network inference challenge, BEELINE, Beeline, CausalBench
Full nameGene regulatory network inference family: the DREAM5 network inference challenge, BEELINE and CausalBench
Ambiguous names (not auto-linked)DREAM, DREAM challenges
FamilyThree different protocols: DREAM5 is a 2010 blind challenge, BEELINE a single-cell evaluation framework and CausalBench a CRISPRi perturbation causal suite; scores do not compare across members.
  • DREAM5 network inference challenge · †
  • DREAM5 results paper ·
  • BEELINE ·
  • CausalBench ·
DomainGenomics & RNA (also: Single-cell & perturbation)
InclusionOriginal 30
TypeA B C · A closed blind challenge plus an evaluation framework and a perturbation suite
Evidence levelL2–L5 · Mostly retrospective comparison; one DREAM5 experimental check reaches L5
PublisherDREAM Challenges, Virginia Tech, GSK.ai (Mixed)
First release† partly verified
Basis: Earliest family member: the year of the DREAM5 challenge; the conference was in November 2010 and the launch month is unsourced source
Other milestones
  • Preprint v1: † · BEELINE bioRxiv v1 source
  • Journal online: · DREAM5 results paper, Nature Methods online source
  • Challenge opens: † · Year of the DREAM5 challenge source
Latest update† partly verified · paper revision
Basis: CausalBench published in Communications Biology
CausalBench was published in Communications Biology; its arXiv v1 is from 2022-10-31 and the challenge ran in 2023.
Note: The site note cites a 2025-05 PMLR version; only the journal version was verified this round. source
The most recent sourced event, not a claim that nothing changed since.
Next expectedNone of the three members has announced a new round or version.
Status◐ Low-frequency · DREAM5 frozen; BEELINE lightly maintained; the CausalBench challenge ended in 2023
ScaleDREAM5: 4 networks, 35 methods; BEELINE: 12 algorithms, 400+ simulated datasets; CausalBench: 2 CRISPRi datasets, over 200,000 samples.
  • Networks: 4 (DREAM5)
  • Methods: 29 submissions plus 6 off-the-shelf tools: 35 (DREAM5)
  • Algorithms: 12 (BEELINE)
  • Simulated datasets: 400+ (BEELINE)
  • Experimental scRNA-seq datasets: 5 (BEELINE)
  • CRISPRi datasets, K562 and RPE1: 2 (CausalBench)
  • Interventional samples: 200000+ cells (CausalBench)
Wet-lab refreshStatic data · No: DREAM5 once tested 53 new predictions and 23 were supported; nothing has been added since.
Own wet lab needed to evaluate a new modelNo, run it as a download
Models reported0 counted (unconfirmed 2)
Unverified fields† first_release, latest_update, ranking.entries[0], ranking.entries[1], ranking.entries[2], ranking.snapshot_date, release_milestones.challenge_open
As of2026-11-16

What it measures

  • Task: Given expression data, infer directed regulator-to-target edges; unsupervised, with no train/test split.
  • Truth: Known networks and pathway libraries: GeneNetWeaver simulations, RegulonDB, ChIP plus motifs, and interaction libraries such as STRING and CORUM.
  • Primary metric: Combined AUPR/AUROC score in DREAM5
  • Secondary metrics: AUPRC ratio and EPR in BEELINE; Mean Wasserstein and false omission rate in CausalBench; Precision and recall against pathway libraries
  • Version traps: State which generation, dataset and gold-standard library: BEELINE data are public and tunable while DREAM5 was blind, so the two do not mix.

Current ranking

Not comparable

MetricDREAM5: combined AUPR/AUROC; BEELINE: AUPRC ratio and EPR
Eval setDREAM5 network inference (frozen), BEELINE and CausalBench suites
Snapshot† partly verified stale
Boardhttps://dreamchallenges.org/dream-5-network-inference-challenge/
Retrieved2026-11-16

Caveat. DREAM5 has been frozen since 2012 with no site model entered, BEELINE and CausalBench have no live board, and foundation-model results use their own splits, so no comparable ranking exists.

No headline entry: Protocols are not comparable and no entry carries a displayable score.

The entries below use different protocols, so they are listed, not ranked, and the overview table shows no score.

RankModelScore (verbatim)Metric · eval set / protocolSource type · dateSource and note
—Community (consensus) networkWithheld: Protocols are not comparable and the value is unverified, so no score is shown.DREAM5: combined AUPR/AUROC; BEELINE: AUPRC ratio and EPR
DREAM5 network inference (frozen), BEELINE and CausalBench suites
Board
source Not a site model; averaging edge ranks across methods gave a consensus network that robustly beat any single method. No value was re-verified, so no score is shown.
—scGPT / Geneformer, attention-edge GRN inferenceWithheld: The source gives no number and is a preprint without peer review.DREAM5: combined AUPR/AUROC; BEELINE: AUPRC ratio and EPR
DREAM5 network inference (frozen), BEELINE and CausalBench suites
Third-party
source A third-party preprint using the BEELINE protocol: attention edges mostly reflect co-expression, with no numeric score.
—scPRINT-2Withheld: A different protocol, with no citable number.DREAM5: combined AUPR/AUROC; BEELINE: AUPRC ratio and EPR
DREAM5 network inference (frozen), BEELINE and CausalBench suites
Self-reported
source scPRINT-2 self-reports gains on its own GRN benchmark rather than the DREAM5 or BEELINE protocol, and no number was located this round.

Models reported

Grouped by relation type; each line shows its source type. Counted relations add up to 0. A pairing whose evidence rests only on this survey's own page is shown as unverified.

Unconfirmed (2, not counted)

  • scGPT / Geneformer Unconfirmed source · A third-party attention-edge study, read in abstract only. low confidence †disputed · evidence: link
  • scPRINT-2 Unconfirmed source · The preprint mentions its own GRN benchmark with scores in the body, not under this family's protocol. †disputed · evidence: link

Caveats

  1. Gold standards contain only known edges, so a predicted false positive may be a real unknown edge and scores mainly reflect truth quality.
  2. Most methods are only slightly better than random; on BEELINE's experimental data most EPR values sit near 1 and none exceeds 1.5.
  3. CausalBench found interventional methods no better than observational ones, contradicting synthetic benchmarks.
  4. Foundation models pretrain on these public datasets and ChIP libraries, and the contamination risk has not been systematically assessed.
  5. The only wet-lab translation is DREAM5's 53 predictions, of which 23 were supported, with no later evidence.

How to use it

Run it statically with no wet lab: BEELINE ships Docker images and configs, so run one curated dataset and compare AUPRC ratios, always alongside a co-expression or correlation baseline, and use CausalBench to test scalability of causal methods. Proving an edge is real regulation needs a CRISPRi or ChIP spot check; do not report AUROC alone.

Pairs well with: Virtual Cell Challenge and perturbation benchmarks · scIB (+Open Problems) · Basenji/Enformer test set

Sources

  1. DREAM5 results paper in Nature Methods (journal) ·
  2. Official DREAM5 challenge page (official site)
  3. BEELINE full text (journal)
  4. BEELINE journal version (journal)
  5. CausalBench paper (preprint) ·
  6. CausalBench journal version (journal) · †

Corrections

4 corrections
FieldWasNowBasisFound
first_release20122010 (the DREAM5 challenge year; 2012-07-15 is the results paper)source2026-11-16
latest_updateCausalBench PMLR paper, 2025-052025-03-11, CausalBench published in Communications Biology (kind: paper-revision)source2026-11-16
familythree generations described in prose onlyexplicit family entry with four members and a boundary statementsource2026-11-16
models_reportedscPRINT-2 counted as reporting this familydisputed-or-unconfirmed: its own GRN benchmark, not the DREAM5/BEELINE protocolsource2026-11-16

Part of the AI for Bio benchmarks survey · all 44 benchmarks · open in the explorer. Record as of 2026-11-16.