Overview
| Name | DREAM5 / BEELINE / CausalBench · aliases: DREAM5, DREAM5 network inference challenge, BEELINE, Beeline, CausalBench |
|---|---|
| Full name | Gene regulatory network inference family: the DREAM5 network inference challenge, BEELINE and CausalBench |
| Ambiguous names (not auto-linked) | DREAM, DREAM challenges |
| Family | Three different protocols: DREAM5 is a 2010 blind challenge, BEELINE a single-cell evaluation framework and CausalBench a CRISPRi perturbation causal suite; scores do not compare across members.
|
| Domain | Genomics & RNA (also: Single-cell & perturbation) |
| Inclusion | Original 30 |
| Type | A B C · A closed blind challenge plus an evaluation framework and a perturbation suite |
| Evidence level | L2–L5 · Mostly retrospective comparison; one DREAM5 experimental check reaches L5 |
| Publisher | DREAM Challenges, Virginia Tech, GSK.ai (Mixed) |
| First release | † partly verified Basis: Earliest family member: the year of the DREAM5 challenge; the conference was in November 2010 and the launch month is unsourced source |
| Other milestones | |
| Latest update | † partly verified · paper revision Basis: CausalBench published in Communications Biology CausalBench was published in Communications Biology; its arXiv v1 is from 2022-10-31 and the challenge ran in 2023. Note: The site note cites a 2025-05 PMLR version; only the journal version was verified this round. source The most recent sourced event, not a claim that nothing changed since. |
| Next expected | None of the three members has announced a new round or version. |
| Status | ◐ Low-frequency · DREAM5 frozen; BEELINE lightly maintained; the CausalBench challenge ended in 2023 |
| Scale | DREAM5: 4 networks, 35 methods; BEELINE: 12 algorithms, 400+ simulated datasets; CausalBench: 2 CRISPRi datasets, over 200,000 samples.
|
| Wet-lab refresh | Static data · No: DREAM5 once tested 53 new predictions and 23 were supported; nothing has been added since. |
| Own wet lab needed to evaluate a new model | No, run it as a download |
| Models reported | 0 counted (unconfirmed 2) |
| Unverified fields | † first_release, latest_update, ranking.entries[0], ranking.entries[1], ranking.entries[2], ranking.snapshot_date, release_milestones.challenge_open |
| As of | 2026-11-16 |
What it measures
- Task: Given expression data, infer directed regulator-to-target edges; unsupervised, with no train/test split.
- Truth: Known networks and pathway libraries: GeneNetWeaver simulations, RegulonDB, ChIP plus motifs, and interaction libraries such as STRING and CORUM.
- Primary metric: Combined AUPR/AUROC score in DREAM5
- Secondary metrics: AUPRC ratio and EPR in BEELINE; Mean Wasserstein and false omission rate in CausalBench; Precision and recall against pathway libraries
- Version traps: State which generation, dataset and gold-standard library: BEELINE data are public and tunable while DREAM5 was blind, so the two do not mix.
Current ranking
Not comparable
| Metric | DREAM5: combined AUPR/AUROC; BEELINE: AUPRC ratio and EPR |
|---|---|
| Eval set | DREAM5 network inference (frozen), BEELINE and CausalBench suites |
| Snapshot | † partly verified stale |
| Board | https://dreamchallenges.org/dream-5-network-inference-challenge/ |
| Retrieved | 2026-11-16 |
Caveat. DREAM5 has been frozen since 2012 with no site model entered, BEELINE and CausalBench have no live board, and foundation-model results use their own splits, so no comparable ranking exists.
No headline entry: Protocols are not comparable and no entry carries a displayable score.
The entries below use different protocols, so they are listed, not ranked, and the overview table shows no score.
| Rank | Model | Score (verbatim) | Metric · eval set / protocol | Source type · date | Source and note |
|---|---|---|---|---|---|
| — | Community (consensus) network | Withheld: Protocols are not comparable and the value is unverified, so no score is shown. | DREAM5: combined AUPR/AUROC; BEELINE: AUPRC ratio and EPR DREAM5 network inference (frozen), BEELINE and CausalBench suites | Board | source Not a site model; averaging edge ranks across methods gave a consensus network that robustly beat any single method. No value was re-verified, so no score is shown. |
| — | scGPT / Geneformer, attention-edge GRN inference | Withheld: The source gives no number and is a preprint without peer review. | DREAM5: combined AUPR/AUROC; BEELINE: AUPRC ratio and EPR DREAM5 network inference (frozen), BEELINE and CausalBench suites | Third-party | source A third-party preprint using the BEELINE protocol: attention edges mostly reflect co-expression, with no numeric score. |
| — | scPRINT-2 | Withheld: A different protocol, with no citable number. | DREAM5: combined AUPR/AUROC; BEELINE: AUPRC ratio and EPR DREAM5 network inference (frozen), BEELINE and CausalBench suites | Self-reported | source scPRINT-2 self-reports gains on its own GRN benchmark rather than the DREAM5 or BEELINE protocol, and no number was located this round. |
Models reported
Grouped by relation type; each line shows its source type. Counted relations add up to 0. A pairing whose evidence rests only on this survey's own page is shown as unverified.
Unconfirmed (2, not counted)
- scGPT / Geneformer Unconfirmed source · A third-party attention-edge study, read in abstract only. low confidence †disputed · evidence: link
- scPRINT-2 Unconfirmed source · The preprint mentions its own GRN benchmark with scores in the body, not under this family's protocol. †disputed · evidence: link
Caveats
- Gold standards contain only known edges, so a predicted false positive may be a real unknown edge and scores mainly reflect truth quality.
- Most methods are only slightly better than random; on BEELINE's experimental data most EPR values sit near 1 and none exceeds 1.5.
- CausalBench found interventional methods no better than observational ones, contradicting synthetic benchmarks.
- Foundation models pretrain on these public datasets and ChIP libraries, and the contamination risk has not been systematically assessed.
- The only wet-lab translation is DREAM5's 53 predictions, of which 23 were supported, with no later evidence.
How to use it
Run it statically with no wet lab: BEELINE ships Docker images and configs, so run one curated dataset and compare AUPRC ratios, always alongside a co-expression or correlation baseline, and use CausalBench to test scalability of causal methods. Proving an edge is real regulation needs a CRISPRi or ChIP spot check; do not report AUROC alone.
Pairs well with: Virtual Cell Challenge and perturbation benchmarks · scIB (+Open Problems) · Basenji/Enformer test set
Sources
- DREAM5 results paper in Nature Methods (journal) ·
- Official DREAM5 challenge page (official site)
- BEELINE full text (journal)
- BEELINE journal version (journal)
- CausalBench paper (preprint) ·
- CausalBench journal version (journal) · †
Corrections
4 corrections
| Field | Was | Now | Basis | Found |
|---|---|---|---|---|
| first_release | 2012 | 2010 (the DREAM5 challenge year; 2012-07-15 is the results paper) | source | 2026-11-16 |
| latest_update | CausalBench PMLR paper, 2025-05 | 2025-03-11, CausalBench published in Communications Biology (kind: paper-revision) | source | 2026-11-16 |
| family | three generations described in prose only | explicit family entry with four members and a boundary statement | source | 2026-11-16 |
| models_reported | scPRINT-2 counted as reporting this family | disputed-or-unconfirmed: its own GRN benchmark, not the DREAM5/BEELINE protocol | source | 2026-11-16 |
Part of the AI for Bio benchmarks survey · all 44 benchmarks · open in the explorer. Record as of 2026-11-16.