Overview
| Name | GUE + Nucleotide Transformer benchmark · aliases: GUE, GUE+, Genome Understanding Evaluation, NT benchmark, Nucleotide Transformer downstream tasks, Genomic Benchmarks, NTv3 Benchmark |
|---|---|
| Full name | GUE (Genome Understanding Evaluation), Nucleotide Transformer downstream tasks and Genomic Benchmarks |
| Ambiguous names (not auto-linked) | Nucleotide Transformer, NT |
| Family | Family entry: GUE/GUE+, NT downstream tasks (original and revised not comparable), Genomic Benchmarks and NTv3 Benchmark; BEND and DART-Eval are separate.
|
| Domain | Genomics & RNA |
| Inclusion | Original 30 |
| Type | B · Static dataset suite |
| Evidence level | L2–L3 · Revised NT (chromosome hold-out) near L3 |
| Publisher | InstaDeep, Northwestern University, CEITEC, Masaryk University (Mixed) |
| First release | · paper v1 Basis: Earliest family member: Genomic Benchmarks bioRxiv v1 (journal version BMC Genomic Data 2023-05-01) source |
| Other milestones |
|
| Latest update | · data Basis: NTv3 preprint bioRxiv v1 NTv3 Benchmark (106 tasks, 7 species) released with the NTv3 preprint (bioRxiv v1 2025-12-25); an InstaDeep blog of 2026-02-18 later announced the same release. source The most recent sourced event, not a claim that nothing changed since. |
| Next expected | No next version announced for any suite. |
| Status | ◐ Low-frequency · Still used but saturated; the larger GENEB suite (2026-06) partly replaces it |
| Scale | GUE: 28 datasets (+8 GUE+); NT: 18 tasks, 532,064 rows in the revised version; Genomic Benchmarks: 9 datasets.
|
| Wet-lab refresh | Static data · None; reuses public data such as ENCODE. |
| Own wet lab needed to evaluate a new model | No, run it as a download |
| Models reported | 0 counted |
| Unverified fields | † family.members[2].first_release, family.members[3].first_release, release_milestones.journal_online, ranking.caveat |
| As of | 2026-11-16 |
What it measures
- Task: Whether a DNA language model can classify short (up to 1 kb) sequences for regulatory features such as promoters, enhancers, splice sites and histone marks.
- Truth: Element labels from public annotations such as ENCODE; negatives differ by version (shuffled, random or real genomic sequence).
- Primary metric: MCC
- Secondary metrics: F1 (GUE virus-variant task); 10-fold CV MCC (revised NT)
- Version traps: Original and revised NT scores are not comparable; tokenizer, fine-tuning vs frozen probe and pooling all change scores and must be stated.
Current ranking
No ranking
| Metric | MCC (GUE average over 28 datasets; NT 10-fold CV MCC) |
|---|---|
| Eval set | GUE / NT downstream (original & revised) / Genomic Benchmarks |
| Snapshot | No sourced date† |
| Board | No official board |
| Retrieved | 2026-11-16 |
Caveat. No official board and no site model reports on these suites (NTv3 uses its own 106-task benchmark; Evo 2 does not report). Two non-site models differ by 0.13 GUE average MCC, within seed noise (not re-verified).
No headline entry: No official board and no site model reports on these suites, so there is no headline number.
No ranking entries recorded.
Models reported
Grouped by relation type; each line shows its source type. Counted relations add up to 0. A pairing whose evidence rests only on this survey's own page is shown as unverified.
None. No model among the 112 has a recorded pairing with this entry.
Caveats
- Randomly initialised same-architecture models and small CNNs often match or beat pretrained models; seed noise reaches 5 points; original and revised NT versions are not comparable.
- Mostly measures local motifs and GC content; only in-distribution short-sequence classification, with no variant, cross-cell-type or cross-species OOD and no prospective test.
- NTv3 and Evo 2 do not report on the GUE/NT suites; the main users are not site models.
- The GUE paper gives inconsistent dataset counts (28, 36 and others); this page uses 28 + 8.
How to use it
Download and run; no wet lab needed. Use the revised NT version; report confidence intervals over several seeds; include randomly initialised same-architecture, small-CNN and k-mer-plus-linear baselines, and state tokenizer and fine-tuning or probing setup. Add BEND (long-range and variants); for variants see ClinVar and Enformer.
Pairs well with: BEND · ClinVar hold-out protocol · Basenji/Enformer test set · TraitGym
Sources
- GUE / DNABERT-2 paper, arXiv 2306.15006 (preprint)
- Nucleotide Transformer paper, bioRxiv (preprint)
- Nucleotide Transformer journal version (Nature Methods) (journal)
- Revised NT dataset card (repository)
- Genomic Benchmarks, bioRxiv (preprint)
- genomic-benchmarks on PyPI (package registry)
- NTv3 preprint (includes NTv3 Benchmark) (preprint)
- GENEB suite, arXiv 2606.04525 (third-party paper)
Corrections
5 corrections
| Field | Was | Now | Basis | Found |
|---|---|---|---|---|
| latest_update | 2025-07-30 | 2025-12-25 | source | 2026-11-16 |
| latest_update_desc | Genomic Benchmarks repo: cache moved to Zenodo and version bump; NT revised HF dataset got 'task' column 2025-06-30. | NTv3 Benchmark (106 tasks, 7 species) released with the Nucleotide Transformer v3 preprint (bioRxiv v1 2025-12-25); InstaDeep blog 2026-02-18 is a later announcement of the same release. | source | 2026-11-16 |
| latest_update_kind | code-maintenance-only | data-release | source | 2026-11-16 |
| family.members[GUE].first_release | 2023-06-27 (arXiv 2306.15794) | 2023-06 (arXiv 2306.15006; day unverified) | source | 2026-11-16 |
| publisher (NT journal date) | Nature Methods 2025-02 | Nature Methods online 2024-11-28 | source | 2026-11-16 |
Part of the AI for Bio benchmarks survey · all 44 benchmarks · open in the explorer. Record as of 2026-11-16.