Overview
| Name | TraitGym · aliases: TraitGym |
|---|---|
| Full name | The authors use only the name TraitGym; no expanded form exists. |
| Ambiguous names (not auto-linked) | None |
| Family | Not a family entry. The two subsets are configurations of one dataset, not a family. |
| Domain | Genomics & RNA |
| Inclusion | Supplementary entry |
| Type | B · Fixed variant set plus a public leaderboard. |
| Evidence level | L3 · Retrospective evaluation with carefully matched controls; not blind. |
| Publisher | University of California, Berkeley (Academic lab) |
| First release | · paper v1 Basis: bioRxiv preprint v1 (10.1101/2025.02.11.637758); Hugging Face dataset and leaderboard creation dates are not used as first-release evidence source |
| Other milestones | |
| Latest update | † partly verified · maintenance only Basis: commit date of GitHub tag 0.1; since then main has only a CI commit Code tag 0.1; since then main has only a CI commit (2026-01-19) plus unmerged revision branches; paper v2 appeared on 2025-03-04. Note: No release object was found; the date is the tag's commit date. source The most recent sourced event, not a claim that nothing changed since. |
| Next expected | The authors have announced no new version; the revision branches carry no announced date. |
| Status | ◐ Low-frequency · The leaderboard is still online but has had no new entries since 2025-03; the paper is still a preprint. |
| Scale | 338 Mendelian variants (113 traits); 1,140 complex-trait variants (83 traits); nine matched controls each.
|
| Wet-lab refresh | Static data · Reuses public OMIM and UK Biobank fine-mapping results; no variants are added. |
| Own wet lab needed to evaluate a new model | No, run it as a download |
| Models reported | 3 counted (modified protocol 1, retrospective 2) |
| Unverified fields | † latest_update, ranking.snapshot_date, models_reported[0] |
| As of | 2026-11-16 |
What it measures
- Task: Binary classification separating known or strong-candidate causal non-coding regulatory variants from controls matched on position, consequence and related covariates.
- Truth: Mendelian positives come from OMIM causal variants; complex-trait positives come from statistical fine-mapping candidates.
- Primary metric: AUPRC weighted-averaged by chromosome.
- Secondary metrics: Bootstrap standard error.
- Version traps: State the subset (Mendelian or complex traits), the configuration (matched_9 or all) and whether the track is zero-shot or linear probing; standings are not interchangeable.
Current ranking
Board
| Metric | AUPRC weighted-averaged by chromosome (main metric), with standard error |
|---|---|
| Eval set | TraitGym Mendelian traits (matched_9) and complex traits (matched_9), zero-shot and linear-probing tracks |
| Snapshot | stale |
| Board | https://huggingface.co/spaces/songlab/TraitGym-leaderboard |
| Retrieved | 2026-11-16 |
Caveat. The board snapshot is 2025-03-07 with no models added since; ranks are per subset and track and cannot be compared across slices; values keep the board's own rounding.
Shown in the overview table: Board
| Rank | Model | Score (verbatim) | Metric · eval set / protocol | Source type · date | Source and note |
|---|---|---|---|---|---|
| 1 | CADD | 0.712 (se 0.038) | AUPRC weighted-averaged by chromosome (main metric), with standard error TraitGym Mendelian traits (matched_9) and Complex traits (matched_9), subset 'all', zero-shot and linear-probing tracks | Board | source Not a site model; first on the Mendelian set, zero-shot. |
| 5 | Evo2 40B | 0.557 (se 0.051) | AUPRC weighted-averaged by chromosome (main metric), with standard error TraitGym Mendelian traits (matched_9) and Complex traits (matched_9), subset 'all', zero-shot and linear-probing tracks | Board | source Mendelian set, zero-shot; the 7B checkpoint scores 0.385 (rank 10) and the 1B 0.254 (rank 13). |
| 7 | Borzoi | 0.436 (se 0.052) | AUPRC weighted-averaged by chromosome (main metric), with standard error TraitGym Mendelian traits (matched_9) and Complex traits (matched_9), subset 'all', zero-shot and linear-probing tracks | Board | source Mendelian set, zero-shot. |
| 1 | Enformer | 0.303 (se 0.010) | AUPRC weighted-averaged by chromosome (main metric), with standard error TraitGym Mendelian traits (matched_9) and Complex traits (matched_9), subset 'all', zero-shot and linear-probing tracks | Board | source No site model page; first on complex traits with linear probing, where the site has only the same-named benchmark entry. |
| 2 | Borzoi | 0.297 (se 0.014) | AUPRC weighted-averaged by chromosome (main metric), with standard error TraitGym Mendelian traits (matched_9) and Complex traits (matched_9), subset 'all', zero-shot and linear-probing tracks | Board | source Complex traits, linear probing; the best standing of any site model on the board. |
| 9 | Evo2 40B | 0.153 (se 0.007) | AUPRC weighted-averaged by chromosome (main metric), with standard error TraitGym Mendelian traits (matched_9) and Complex traits (matched_9), subset 'all', zero-shot and linear-probing tracks | Board | source Complex traits, linear probing. |
Models reported
Grouped by relation type; each line shows its source type. Counted relations add up to 3. A pairing whose evidence rests only on this survey's own page is shown as unverified.
Developer report, modified protocol (1)
- AlphaGenome Self-reported · Uses the variant set and matched controls but reports enrichment rather than AUPRC, so it is not board-comparable. †partly verified · evidence: link
Retrospective dataset use (2)
- Borzoi Third-party · Evaluated on this set by the benchmark's authors. †partly verified · evidence: link
- Evo 2 Third-party · Evaluated on this set by the benchmark's authors. · evidence: link
Caveats
- Complex-trait positives come from statistical fine-mapping, so causality is probabilistic rather than experimentally confirmed.
- The Mendelian set has only 338 positives unevenly spread across traits, so small gaps are not a ranking.
- Not blind: variants and controls come from public data that may sit inside model training corpora.
- Results are stratified: alignment-based methods lead on Mendelian and complex disease traits, while track-prediction models do better on complex non-disease traits.
- The board has had no entries since 2025-03, and AlphaGenome is absent from it, using a different protocol.
How to use it
Take the data from Hugging Face and score with the provided scripts. Report chromosome-weighted AUPRC with standard errors separately for the Mendelian and complex-trait matched_9 configurations; table alignment-based methods alongside and give the gap to an ensemble baseline; split by variant class so one class cannot dominate; state whether the track is zero-shot or linear probing. Using the variant set is not the same as entering the board, and new-trait claims need independent functional validation or a prospective variant set.
Pairs well with: Basenji/Enformer test set · ClinVar hold-out protocol · GUE + Nucleotide Transformer benchmark · BEND · CAGI
Sources
- TraitGym preprint (bioRxiv 10.1101/2025.02.11.637758) (preprint) ·
- TraitGym preprint v2 (preprint) ·
- Official repository songlab-cal/TraitGym (repository)
- Hugging Face dataset (package registry)
- Hugging Face leaderboard (leaderboard) ·
- AlphaGenome paper (Nature 2026, PMC12851941) (journal)
Corrections
3 corrections
Part of the AI for Bio benchmarks survey · all 44 benchmarks · open in the explorer. Record as of 2026-11-16.