AI for Bio · Benchmark 16

GUE + Nucleotide Transformer benchmark

Short-sequence regulatory-element classification suites for DNA language models; a smoke test, not proof of quality.

bench-16Type BEvidence L2–L3Genomics & RNA◐ Low-frequency No ranking中文

Overview

NameGUE + Nucleotide Transformer benchmark · aliases: GUE, GUE+, Genome Understanding Evaluation, NT benchmark, Nucleotide Transformer downstream tasks, Genomic Benchmarks, NTv3 Benchmark
Full nameGUE (Genome Understanding Evaluation), Nucleotide Transformer downstream tasks and Genomic Benchmarks
Ambiguous names (not auto-linked)Nucleotide Transformer, NT
FamilyFamily entry: GUE/GUE+, NT downstream tasks (original and revised not comparable), Genomic Benchmarks and NTv3 Benchmark; BEND and DART-Eval are separate.
  • Genomic Benchmarks ·
  • Nucleotide Transformer downstream tasks (original) ·
  • GUE / GUE+ (DNABERT-2) · †
  • Nucleotide Transformer downstream tasks (revised) · c. 2024 †
  • NTv3 Benchmark ·
DomainGenomics & RNA
InclusionOriginal 30
TypeB · Static dataset suite
Evidence levelL2–L3 · Revised NT (chromosome hold-out) near L3
PublisherInstaDeep, Northwestern University, CEITEC, Masaryk University (Mixed)
First release · paper v1
Basis: Earliest family member: Genomic Benchmarks bioRxiv v1 (journal version BMC Genomic Data 2023-05-01) source
Other milestones
  • Preprint v1: · Earliest family member: Genomic Benchmarks bioRxiv v1 (journal version BMC Genomic Data 2023-05-01) source
  • Journal online: † · Genomic Benchmarks journal version (BMC Genomic Data)
Latest update · data
Basis: NTv3 preprint bioRxiv v1
NTv3 Benchmark (106 tasks, 7 species) released with the NTv3 preprint (bioRxiv v1 2025-12-25); an InstaDeep blog of 2026-02-18 later announced the same release. source
The most recent sourced event, not a claim that nothing changed since.
Next expectedNo next version announced for any suite.
Status◐ Low-frequency · Still used but saturated; the larger GENEB suite (2026-06) partly replaces it
ScaleGUE: 28 datasets (+8 GUE+); NT: 18 tasks, 532,064 rows in the revised version; Genomic Benchmarks: 9 datasets.
  • Datasets: 28 + 8 (GUE + GUE+)
  • Tasks: 18 (NT downstream)
  • Rows: 532,064 (NT revised)
  • Datasets: 9 (Genomic Benchmarks)
  • Tasks (7 species): 106 (NTv3 Benchmark)
Wet-lab refreshStatic data · None; reuses public data such as ENCODE.
Own wet lab needed to evaluate a new modelNo, run it as a download
Models reported0 counted
Unverified fields† family.members[2].first_release, family.members[3].first_release, release_milestones.journal_online, ranking.caveat
As of2026-11-16

What it measures

  • Task: Whether a DNA language model can classify short (up to 1 kb) sequences for regulatory features such as promoters, enhancers, splice sites and histone marks.
  • Truth: Element labels from public annotations such as ENCODE; negatives differ by version (shuffled, random or real genomic sequence).
  • Primary metric: MCC
  • Secondary metrics: F1 (GUE virus-variant task); 10-fold CV MCC (revised NT)
  • Version traps: Original and revised NT scores are not comparable; tokenizer, fine-tuning vs frozen probe and pooling all change scores and must be stated.

Current ranking

No ranking

MetricMCC (GUE average over 28 datasets; NT 10-fold CV MCC)
Eval setGUE / NT downstream (original & revised) / Genomic Benchmarks
SnapshotNo sourced date†
BoardNo official board
Retrieved2026-11-16

Caveat. No official board and no site model reports on these suites (NTv3 uses its own 106-task benchmark; Evo 2 does not report). Two non-site models differ by 0.13 GUE average MCC, within seed noise (not re-verified).

No headline entry: No official board and no site model reports on these suites, so there is no headline number.

No ranking entries recorded.

Models reported

Grouped by relation type; each line shows its source type. Counted relations add up to 0. A pairing whose evidence rests only on this survey's own page is shown as unverified.

None. No model among the 112 has a recorded pairing with this entry.

Caveats

  1. Randomly initialised same-architecture models and small CNNs often match or beat pretrained models; seed noise reaches 5 points; original and revised NT versions are not comparable.
  2. Mostly measures local motifs and GC content; only in-distribution short-sequence classification, with no variant, cross-cell-type or cross-species OOD and no prospective test.
  3. NTv3 and Evo 2 do not report on the GUE/NT suites; the main users are not site models.
  4. The GUE paper gives inconsistent dataset counts (28, 36 and others); this page uses 28 + 8.

How to use it

Download and run; no wet lab needed. Use the revised NT version; report confidence intervals over several seeds; include randomly initialised same-architecture, small-CNN and k-mer-plus-linear baselines, and state tokenizer and fine-tuning or probing setup. Add BEND (long-range and variants); for variants see ClinVar and Enformer.

Pairs well with: BEND · ClinVar hold-out protocol · Basenji/Enformer test set · TraitGym

Sources

  1. GUE / DNABERT-2 paper, arXiv 2306.15006 (preprint)
  2. Nucleotide Transformer paper, bioRxiv (preprint)
  3. Nucleotide Transformer journal version (Nature Methods) (journal)
  4. Revised NT dataset card (repository)
  5. Genomic Benchmarks, bioRxiv (preprint)
  6. genomic-benchmarks on PyPI (package registry)
  7. NTv3 preprint (includes NTv3 Benchmark) (preprint)
  8. GENEB suite, arXiv 2606.04525 (third-party paper)

Corrections

5 corrections
FieldWasNowBasisFound
latest_update2025-07-302025-12-25source2026-11-16
latest_update_descGenomic Benchmarks repo: cache moved to Zenodo and version bump; NT revised HF dataset got 'task' column 2025-06-30.NTv3 Benchmark (106 tasks, 7 species) released with the Nucleotide Transformer v3 preprint (bioRxiv v1 2025-12-25); InstaDeep blog 2026-02-18 is a later announcement of the same release.source2026-11-16
latest_update_kindcode-maintenance-onlydata-releasesource2026-11-16
family.members[GUE].first_release2023-06-27 (arXiv 2306.15794)2023-06 (arXiv 2306.15006; day unverified)source2026-11-16
publisher (NT journal date)Nature Methods 2025-02Nature Methods online 2024-11-28source2026-11-16

Part of the AI for Bio benchmarks survey · all 44 benchmarks · open in the explorer. Record as of 2026-11-16.