AI for Bio · Benchmark 08

Basenji/Enformer test set

The de facto protocol for sequence-to-function models: track prediction on fixed held-out regions.

bench-08Type BEvidence L2–L3Genomics & RNA● Active Self-reported中文

Overview

NameBasenji/Enformer test set · aliases: Basenji test set, Basenji2 test split, Enformer test set, cross2020 data
Full nameThe Basenji/Enformer sequence-to-function test set: the Basenji2 data and its fixed splits
Ambiguous names (not auto-linked)Enformer, Basenji
FamilyThis entry covers the data and its fixed splits, not any model; later models re-fold or change track sets, which counts as their own protocol.
  • Basenji ·
  • Basenji2 cross-species data · †
  • Enformer ·
DomainGenomics & RNA
InclusionOriginal 30
TypeB · Static dataset suite with fixed splits and no leaderboard
Evidence levelL2–L3 · Main task holds out regions of one genome; the variant task is near L3
PublisherCalico Life Sciences, DeepMind (Mixed)
First release† partly verified · paper v1
Basis: Earliest family member: the Basenji bioRxiv preprint, journal version 2018-03-27; the fixed splits came later with the Basenji2 data.
Note: Family-level date, not the release of the later model of the same name source
Other milestones
  • First public evidence: · Earliest public preprint in the family source
  • Preprint v1: · The preprint that established the protocol in common use today source
  • Journal online: · Journal online publication source
Latest update† partly verified · maintenance only
Basis: Commit history of the official repository holding the earlier data
A dependency-freezing commit in the earlier data repository: code maintenance only, with the benchmark data and splits unchanged.
Note: The commit is in the earlier data repository, not in a later model's repository source
The most recent sourced event, not a claim that nothing changed since.
Next expectedNo maintainer has announced a new version or new splits.
Status● Active · The data are stable and reused by newer models; no leaderboard
Scale5,313 human tracks and 1,643 mouse tracks; the human test split has 1,937 sequences of 393,216 bp.
  • Human tracks: 5313 (Basenji2)
  • Mouse tracks: 1643 (Basenji2)
  • Human train/valid/test sequences: 34021/2213/1937 (Basenji2)
  • Mouse train/valid/test sequences: 29295/2209/2017 (Basenji2)
  • Input sequence length: 393216 bp (Enformer)
  • Output bin width: 128 bp (Enformer)
Wet-lab refreshStatic data · No new experiments; reuses public epigenomic tracks, with public eQTL and MPRA data for variants.
Own wet lab needed to evaluate a new modelYes, to claim real generalization
Models reported3 counted (self 3)
Unverified fields† first_release, latest_update, family.members[1], ranking.entries[1], ranking.entries[2], ranking.entries[0]
As of2026-11-16

What it measures

  • Task: Predict thousands of epigenomic and transcriptional tracks from DNA sequence and score variant effects from reference-versus-alternate differences.
  • Truth: Public experimental tracks, heterogeneous in batch and coverage; variant labels come from statistical fine-mapping, not experiments.
  • Primary metric: Track Pearson correlation at 128 bp bins
  • Secondary metrics: Gene-level Pearson correlation; Variant-effect auROC
  • Version traps: State which data generation, folds and resolution were used: later models re-fold the data, so per-bin numbers do not compare directly.

Current ranking

Self-reported

MetricTrack Pearson (128/32/1 bp bins), gene-level Pearson, variant auROC
Eval setBasenji2 held-out test intervals (Borzoi/AlphaGenome use own re-folded splits)
Snapshot
BoardNo official board
Retrieved2026-11-16

Caveat. There is no leaderboard; each paper uses different track sets, folds and resolutions, so numbers do not compare across papers and relative gains are each paper's own figures.

No headline entry: The sources contain no rank-1 entry and the entries use different protocols, so no headline number is given.

RankModelScore (verbatim)Metric · eval set / protocolSource type · dateSource and note
—AlphaGenome (all-folds, pre-trained)DNase 128bp r=0.87 (+2.4% vs Enformer); CAGE 128bp r=0.71 (-0.3% vs Enformer); RNA-seq 1bp r=0.59 (+28.2% vs Borzoi)Track Pearson (128/32/1 bp bins), gene-level Pearson, variant auROC
Basenji2 held-out test intervals (Borzoi/AlphaGenome use own re-folded splits)
Self-reported
source second-hand Model card checked 2026-11-16; the paper's broader claims of leading were not re-verified
—EnformerCAGE gene-level Pearson 0.85 (Basenji2 0.81; replicate ceiling 0.94)Track Pearson (128/32/1 bp bins), gene-level Pearson, variant auROC
Basenji2 held-out test intervals (Borzoi/AlphaGenome use own re-folded splits)
Self-reported
source second-hand Not a site model; reference baseline from an existing site note, not re-verified
—Borzoigene-level RNA-seq Pearson ~0.87Track Pearson (128/32/1 bp bins), gene-level Pearson, variant auROC
Basenji2 held-out test intervals (Borzoi/AlphaGenome use own re-folded splits)
Self-reported
source second-hand Second-hand figure from an existing site note; different folds and track set
—NTv3No scoreTrack Pearson (128/32/1 bp bins), gene-level Pearson, variant auROC
Basenji2 held-out test intervals (Borzoi/AlphaGenome use own re-folded splits)
Self-reported
source Self-reported on its own task set; no number on this set, so not comparable

Models reported

Grouped by relation type; each line shows its source type. Counted relations add up to 3. A pairing whose evidence rests only on this survey's own page is shown as unverified.

Developer report, official protocol (3)

  • AlphaGenome Self-reported · Self-reported on this data's track test set · evidence: link
  • Borzoi Self-reported · The paper uses its own variant tasks and also reports on this test set · evidence: link
  • NTv3 Self-reported · Self-report in a technical report, fine-tuned on re-folded splits · evidence: link

Caveats

  1. Cross-gene correlation mainly reflects which genes are highly expressed, not the ability to predict across individuals.
  2. An independent study reported a near-zero mean cross-individual correlation, with some genes significantly anticorrelated.
  3. Gains on reference sequence do not carry over to variant-effect prediction, which is weaker in cell-type-specific regions.
  4. Zero-shot and fine-tuned results are often mixed, and reuse usually reports a single correlation with no confidence interval.
  5. Variant-task labels come from statistical fine-mapping, not experimentally validated causal variants.

How to use it

Download the data and fixed splits, predict tracks on held-out regions and score variants from reference-versus-alternate differences; it runs statically with no wet lab of your own. Report the data generation, folds, resolution and track subset, separate zero-shot from fine-tuned results, and give individual-level evaluation separately rather than cross-gene correlation alone. Pair with TraitGym and BEND.

Pairs well with: TraitGym · BEND · GUE + Nucleotide Transformer benchmark · ClinVar hold-out protocol

Sources

  1. Journal paper establishing the protocol in common use (journal)
  2. Earliest preprint in the family (preprint)
  3. Cross-species data paper (journal)
  4. Repository holding the earlier data and code (repository)
  5. Third-party study of individual-level performance (third-party paper)
  6. Later model paper reusing this data (journal)

Corrections

2 corrections
FieldWasNowBasisFound
first_releasestart year 20182017-07-10 at family level (earliest preprint), with the family members and boundary statedsource2026-11-16
latest_update_kindnew-versioncode-maintenance-onlysource2026-11-16

Part of the AI for Bio benchmarks survey · all 44 benchmarks · open in the explorer. Record as of 2026-11-16.