AI for Bio · Benchmark 37

TraitGym

Benchmark classifying causal non-coding variants against matched controls.

bench-37Type BEvidence L3Genomics & RNA◐ Low-frequency Board中文

Overview

NameTraitGym · aliases: TraitGym
Full nameThe authors use only the name TraitGym; no expanded form exists.
Ambiguous names (not auto-linked)None
FamilyNot a family entry. The two subsets are configurations of one dataset, not a family.
DomainGenomics & RNA
InclusionSupplementary entry
TypeB · Fixed variant set plus a public leaderboard.
Evidence levelL3 · Retrospective evaluation with carefully matched controls; not blind.
PublisherUniversity of California, Berkeley (Academic lab)
First release · paper v1
Basis: bioRxiv preprint v1 (10.1101/2025.02.11.637758); Hugging Face dataset and leaderboard creation dates are not used as first-release evidence source
Other milestones
  • First public evidence: No sourced date† · The Hugging Face dataset (2025-01-26) and leaderboard (2025-02-06) creation dates are excluded as first-release evidence. source
  • Preprint v1: · bioRxiv preprint v1 source
  • Code release: · GitHub tag 0.1 (commit date) source
Latest update† partly verified · maintenance only
Basis: commit date of GitHub tag 0.1; since then main has only a CI commit
Code tag 0.1; since then main has only a CI commit (2026-01-19) plus unmerged revision branches; paper v2 appeared on 2025-03-04.
Note: No release object was found; the date is the tag's commit date. source
The most recent sourced event, not a claim that nothing changed since.
Next expectedThe authors have announced no new version; the revision branches carry no announced date.
Status◐ Low-frequency · The leaderboard is still online but has had no new entries since 2025-03; the paper is still a preprint.
Scale338 Mendelian variants (113 traits); 1,140 complex-trait variants (83 traits); nine matched controls each.
  • Mendelian causal variants: 338 (matched_9)
  • Mendelian controls: 3,042 (matched_9)
  • Complex-trait variants: 1,140 (matched_9)
  • Complex-trait controls: 10,260 (matched_9)
  • Traits covered: 113 / 83 (Mendelian / complex)
Wet-lab refreshStatic data · Reuses public OMIM and UK Biobank fine-mapping results; no variants are added.
Own wet lab needed to evaluate a new modelNo, run it as a download
Models reported3 counted (modified protocol 1, retrospective 2)
Unverified fields† latest_update, ranking.snapshot_date, models_reported[0]
As of2026-11-16

What it measures

  • Task: Binary classification separating known or strong-candidate causal non-coding regulatory variants from controls matched on position, consequence and related covariates.
  • Truth: Mendelian positives come from OMIM causal variants; complex-trait positives come from statistical fine-mapping candidates.
  • Primary metric: AUPRC weighted-averaged by chromosome.
  • Secondary metrics: Bootstrap standard error.
  • Version traps: State the subset (Mendelian or complex traits), the configuration (matched_9 or all) and whether the track is zero-shot or linear probing; standings are not interchangeable.

Current ranking

Board

MetricAUPRC weighted-averaged by chromosome (main metric), with standard error
Eval setTraitGym Mendelian traits (matched_9) and complex traits (matched_9), zero-shot and linear-probing tracks
Snapshot stale
Boardhttps://huggingface.co/spaces/songlab/TraitGym-leaderboard
Retrieved2026-11-16

Caveat. The board snapshot is 2025-03-07 with no models added since; ranks are per subset and track and cannot be compared across slices; values keep the board's own rounding.

Shown in the overview table:

CADD
0.712 (se 0.038)
AUPRC weighted-averaged by chromosome (main metric), with standard error · TraitGym Mendelian traits (matched_9) and Complex traits (matched_9), subset 'all', zero-shot and linear-probing tracks
Board · stale
Best site model: Evo2 40B (#5)

RankModelScore (verbatim)Metric · eval set / protocolSource type · dateSource and note
1CADD0.712 (se 0.038)AUPRC weighted-averaged by chromosome (main metric), with standard error
TraitGym Mendelian traits (matched_9) and Complex traits (matched_9), subset 'all', zero-shot and linear-probing tracks
Board
source Not a site model; first on the Mendelian set, zero-shot.
5Evo2 40B0.557 (se 0.051)AUPRC weighted-averaged by chromosome (main metric), with standard error
TraitGym Mendelian traits (matched_9) and Complex traits (matched_9), subset 'all', zero-shot and linear-probing tracks
Board
source Mendelian set, zero-shot; the 7B checkpoint scores 0.385 (rank 10) and the 1B 0.254 (rank 13).
7Borzoi0.436 (se 0.052)AUPRC weighted-averaged by chromosome (main metric), with standard error
TraitGym Mendelian traits (matched_9) and Complex traits (matched_9), subset 'all', zero-shot and linear-probing tracks
Board
source Mendelian set, zero-shot.
1Enformer0.303 (se 0.010)AUPRC weighted-averaged by chromosome (main metric), with standard error
TraitGym Mendelian traits (matched_9) and Complex traits (matched_9), subset 'all', zero-shot and linear-probing tracks
Board
source No site model page; first on complex traits with linear probing, where the site has only the same-named benchmark entry.
2Borzoi0.297 (se 0.014)AUPRC weighted-averaged by chromosome (main metric), with standard error
TraitGym Mendelian traits (matched_9) and Complex traits (matched_9), subset 'all', zero-shot and linear-probing tracks
Board
source Complex traits, linear probing; the best standing of any site model on the board.
9Evo2 40B0.153 (se 0.007)AUPRC weighted-averaged by chromosome (main metric), with standard error
TraitGym Mendelian traits (matched_9) and Complex traits (matched_9), subset 'all', zero-shot and linear-probing tracks
Board
source Complex traits, linear probing.

Models reported

Grouped by relation type; each line shows its source type. Counted relations add up to 3. A pairing whose evidence rests only on this survey's own page is shown as unverified.

Developer report, modified protocol (1)

  • AlphaGenome Self-reported · Uses the variant set and matched controls but reports enrichment rather than AUPRC, so it is not board-comparable. †partly verified · evidence: link

Retrospective dataset use (2)

  • Borzoi Third-party · Evaluated on this set by the benchmark's authors. †partly verified · evidence: link
  • Evo 2 Third-party · Evaluated on this set by the benchmark's authors. · evidence: link

Caveats

  1. Complex-trait positives come from statistical fine-mapping, so causality is probabilistic rather than experimentally confirmed.
  2. The Mendelian set has only 338 positives unevenly spread across traits, so small gaps are not a ranking.
  3. Not blind: variants and controls come from public data that may sit inside model training corpora.
  4. Results are stratified: alignment-based methods lead on Mendelian and complex disease traits, while track-prediction models do better on complex non-disease traits.
  5. The board has had no entries since 2025-03, and AlphaGenome is absent from it, using a different protocol.

How to use it

Take the data from Hugging Face and score with the provided scripts. Report chromosome-weighted AUPRC with standard errors separately for the Mendelian and complex-trait matched_9 configurations; table alignment-based methods alongside and give the gap to an ensemble baseline; split by variant class so one class cannot dominate; state whether the track is zero-shot or linear probing. Using the variant set is not the same as entering the board, and new-trait claims need independent functional validation or a prospective variant set.

Pairs well with: Basenji/Enformer test set · ClinVar hold-out protocol · GUE + Nucleotide Transformer benchmark · BEND · CAGI

Sources

  1. TraitGym preprint (bioRxiv 10.1101/2025.02.11.637758) (preprint) ·
  2. TraitGym preprint v2 (preprint) ·
  3. Official repository songlab-cal/TraitGym (repository)
  4. Hugging Face dataset (package registry)
  5. Hugging Face leaderboard (leaderboard) ·
  6. AlphaGenome paper (Nature 2026, PMC12851941) (journal)

Corrections

3 corrections
FieldWasNowBasisFound
latest_update2025-07-042025-03-25source2026-11-16
latest_update_kindnew-versioncode-maintenance-onlysource2026-11-16
status_noteleaderboard continuously runningleaderboard still online but no new entries since 2025-03source2026-11-16

Part of the AI for Bio benchmarks survey · all 44 benchmarks · open in the explorer. Record as of 2026-11-16.