Overview
| Name | ClinVar hold-out protocol · aliases: ClinVar hold-out protocol, ClinVar benchmark, ClinVar evaluation |
|---|---|
| Full name | Hold-out evaluation protocol built on the clinical variant-label database |
| Ambiguous names (not auto-linked) | ClinVar |
| Family | Not a family entry. This is an evaluation convention around one database, not a family with defined members. |
| Domain | Genomics & RNA |
| Inclusion | Original 30 |
| Type | B · A family of retrospective protocols with no official split |
| Evidence level | L2–L3 · Gene or time splits reach L3; no blind test |
| Publisher | US National Center for Biotechnology Information (Nonprofit institute) |
| First release | † partly verified · data Basis: The official full public release of the database per the official handbook; the November 2012 prototype is listed separately and the exact April day is unsourced. Note: Month only; the evaluation protocol itself has no release date source |
| Other milestones | |
| Latest update | · data Basis: Official FTP release directory, read 2026-11-16 A weekly variant file release and a monthly full-record release; the evaluation protocol itself did not change. source The most recent sourced event, not a claim that nothing changed since. |
| Next expected | The release cadence is observed, not announced with specific dates. |
| Status | ● Active · The database updates monthly; the evaluation protocol has no maintainer |
| Scale | No unified version: each paper builds its own subset, from about ten thousand to over a hundred thousand variants.
|
| Wet-lab refresh | Grows from public sources · No new experiments; the monthly additions are clinical submissions, not new measurements. |
| Own wet lab needed to evaluate a new model | No, run it as a download |
| Models reported | 5 counted (modified protocol 5) |
| Unverified fields | † first_release, release_milestones.first_public_evidence, release_milestones.official_announcement, ranking.snapshot_date, ranking.entries[0], ranking.entries[1], ranking.entries[2], scale.items[1] |
| As of | 2026-11-16 |
What it measures
- Task: Binary classification of held-out variants as pathogenic or benign, mainly missense but also noncoding and splicing.
- Truth: Classifications submitted by clinical laboratories and researchers, carrying review stars, not experimental measurements.
- Primary metric: auROC
- Secondary metrics: auPRC and per-gene mean auROC; Calibrated computational evidence strength
- Version traps: State the snapshot date, star threshold, variant class and source of negatives: changing the negatives moves every method's score.
Current ranking
Not comparable
| Metric | auROC or auPRC, varying by paper |
|---|---|
| Eval set | Each paper builds its own ClinVar subset/snapshot |
| Snapshot | No sourced date† |
| Board | No official board |
| Retrieved | 2026-11-16 |
Caveat. This is a database, not a leaderboard: every paper picks its own snapshot, filtering and variant class, predictors trained on it risk circularity, and no cross-model ranking is possible.
No headline entry: The protocols are not comparable and there is no unified board, so no headline number is given.
The entries below use different protocols, so they are listed, not ranked, and the overview table shows no score.
| Rank | Model | Score (verbatim) | Metric · eval set / protocol | Source type · date | Source and note |
|---|---|---|---|---|---|
| — | AlphaMissense | Withheld: Not comparable: snapshots and subsets differ, so the ranking row shows no score. | auROC or auPRC, varying by paper Each paper builds its own ClinVar subset/snapshot | Self-reported | source second-hand Own missense test set; the figure is corroborated only second-hand, and training on this database inflates it |
| — | AlphaGenome | Withheld: Not comparable: snapshots and subsets differ, so the ranking row shows no score. | auROC or auPRC, varying by paper Each paper builds its own ClinVar subset/snapshot | Self-reported | source second-hand Self-report on a splicing-related subset, from an existing site note, not re-verified |
| — | Evo 2 | Withheld: Not comparable: snapshots and subsets differ, so the ranking row shows no score. | auROC or auPRC, varying by paper Each paper builds its own ClinVar subset/snapshot | Self-reported | source second-hand Figure-only results on its own subsets, with no quotable numbers and mixed standing by variant class |
| — | popEVE | Withheld: Not comparable: snapshots and subsets differ, so the ranking row shows no score. | auROC or auPRC, varying by paper Each paper builds its own ClinVar subset/snapshot | Self-reported | source second-hand Within-gene AUROC in figures and tables, not transcribed; from an existing site note |
| — | BioReason (Evo2+Qwen3-4B) | Withheld: Not comparable: snapshots and subsets differ, so the ranking row shows no score. | auROC or auPRC, varying by paper Each paper builds its own ClinVar subset/snapshot | Self-reported | source second-hand Its own derived split; from an existing site note, not re-verified |
Models reported
Grouped by relation type; each line shows its source type. Counted relations add up to 5. A pairing whose evidence rests only on this survey's own page is shown as unverified.
Developer report, modified protocol (5)
- AlphaGenome Self-reported · Its own variant-effect suite, including splicing classes from this database †partly verified · evidence: link
- AlphaMissense Self-reported · Its own missense test set · evidence: link
- BioReason Self-reported · Its own pathway question task, not this protocol · evidence: link
- Evo 2 Self-reported · Its own subsets, figure-only †partly verified · evidence: link
- popEVE Self-reported · An expert-curated subset from earlier years · evidence: link
Caveats
- There is no official split, predictors and labels contaminate each other, and training on this database inflates scores.
- Pooled coding-missense auROC is already above 0.9, so gaps of 0.01 to 0.03 do not separate methods.
- The gene distribution is heavily skewed, with a few frequently tested genes dominating.
- The truth is submitted clinical interpretation rather than measurement, and submitters sometimes conflict.
- High auROC does not mean clinical usability, which requires calibrated evidence strength.
How to use it
Take a dated snapshot, filter by star level and variant class and score the held-out set; no wet lab of your own is needed. Report the snapshot date, star threshold, source of negatives and gene distribution, and state whether the model has seen these labels. Generalization claims need a time split or held-out genes plus calibrated evidence strength. Pair with ProteinGym and CAGI.
Pairs well with: ProteinGym · CAGI · TraitGym · Basenji/Enformer test set
Sources
- Official database site (official site)
- Database chapter of the official handbook (official site)
- Official release directory (repository)
- Documentation of review status stars (official site)
- Early description paper of the database (journal)
- Model paper reporting this protocol on its own subset (journal)
Corrections
3 corrections
| Field | Was | Now | Basis | Found |
|---|---|---|---|---|
| first_release_basis | ClinVar NAR database paper online (public launch reportedly Apr 2013, not verified) | official full public release of ClinVar, April 2013 (NCBI Handbook); Nov 2012 prototype stored separately, not displayed as first release; exact April day not sourced. | source | 2026-11-16 |
| first_release | 2013-11-14 | 2013-04 (official launch; prototype Nov 2012) | source | 2026-11-16 |
| ranking.entries | per-model scores shown in the ranking row | scores withheld; each entry keeps its protocol description only | source | 2026-11-16 |
Part of the AI for Bio benchmarks survey · all 44 benchmarks · open in the explorer. Record as of 2026-11-16.