AI for Bio · Benchmark 11

ClinVar hold-out protocol

Separating pathogenic from benign on held-out clinical labels, with each paper choosing its own snapshot and subset.

bench-11Type BEvidence L2–L3Genomics & RNA● Active Not comparable中文

Overview

NameClinVar hold-out protocol · aliases: ClinVar hold-out protocol, ClinVar benchmark, ClinVar evaluation
Full nameHold-out evaluation protocol built on the clinical variant-label database
Ambiguous names (not auto-linked)ClinVar
FamilyNot a family entry. This is an evaluation convention around one database, not a family with defined members.
DomainGenomics & RNA
InclusionOriginal 30
TypeB · A family of retrospective protocols with no official split
Evidence levelL2–L3 · Gene or time splits reach L3; no blind test
PublisherUS National Center for Biotechnology Information (Nonprofit institute)
First release† partly verified · data
Basis: The official full public release of the database per the official handbook; the November 2012 prototype is listed separately and the exact April day is unsourced.
Note: Month only; the evaluation protocol itself has no release date source
Other milestones
  • First public evidence: † · Public record of the prototype version source
  • Official announcement: † · Full public release as recorded in the official handbook source
Latest update · data
Basis: Official FTP release directory, read 2026-11-16
A weekly variant file release and a monthly full-record release; the evaluation protocol itself did not change. source
The most recent sourced event, not a claim that nothing changed since.
Next expectedThe release cadence is observed, not announced with specific dates.
Status● Active · The database updates monthly; the evaluation protocol has no maintainer
ScaleNo unified version: each paper builds its own subset, from about ten thousand to over a hundred thousand variants.
  • Test variants in one missense model's own set: 18924 (2023)
  • Missense variants in one protein language model's own set: about 150,000
  • Records in the database, queried 2026-11-16: 4713415 (2026-11-16)
  • Records flagged as pathogenic among them: 266706 (2026-11-16)
Wet-lab refreshGrows from public sources · No new experiments; the monthly additions are clinical submissions, not new measurements.
Own wet lab needed to evaluate a new modelNo, run it as a download
Models reported5 counted (modified protocol 5)
Unverified fields† first_release, release_milestones.first_public_evidence, release_milestones.official_announcement, ranking.snapshot_date, ranking.entries[0], ranking.entries[1], ranking.entries[2], scale.items[1]
As of2026-11-16

What it measures

  • Task: Binary classification of held-out variants as pathogenic or benign, mainly missense but also noncoding and splicing.
  • Truth: Classifications submitted by clinical laboratories and researchers, carrying review stars, not experimental measurements.
  • Primary metric: auROC
  • Secondary metrics: auPRC and per-gene mean auROC; Calibrated computational evidence strength
  • Version traps: State the snapshot date, star threshold, variant class and source of negatives: changing the negatives moves every method's score.

Current ranking

Not comparable

MetricauROC or auPRC, varying by paper
Eval setEach paper builds its own ClinVar subset/snapshot
SnapshotNo sourced date†
BoardNo official board
Retrieved2026-11-16

Caveat. This is a database, not a leaderboard: every paper picks its own snapshot, filtering and variant class, predictors trained on it risk circularity, and no cross-model ranking is possible.

No headline entry: The protocols are not comparable and there is no unified board, so no headline number is given.

The entries below use different protocols, so they are listed, not ranked, and the overview table shows no score.

RankModelScore (verbatim)Metric · eval set / protocolSource type · dateSource and note
—AlphaMissenseWithheld: Not comparable: snapshots and subsets differ, so the ranking row shows no score.auROC or auPRC, varying by paper
Each paper builds its own ClinVar subset/snapshot
Self-reported
source second-hand Own missense test set; the figure is corroborated only second-hand, and training on this database inflates it
—AlphaGenomeWithheld: Not comparable: snapshots and subsets differ, so the ranking row shows no score.auROC or auPRC, varying by paper
Each paper builds its own ClinVar subset/snapshot
Self-reported
source second-hand Self-report on a splicing-related subset, from an existing site note, not re-verified
—Evo 2Withheld: Not comparable: snapshots and subsets differ, so the ranking row shows no score.auROC or auPRC, varying by paper
Each paper builds its own ClinVar subset/snapshot
Self-reported
source second-hand Figure-only results on its own subsets, with no quotable numbers and mixed standing by variant class
—popEVEWithheld: Not comparable: snapshots and subsets differ, so the ranking row shows no score.auROC or auPRC, varying by paper
Each paper builds its own ClinVar subset/snapshot
Self-reported
source second-hand Within-gene AUROC in figures and tables, not transcribed; from an existing site note
—BioReason (Evo2+Qwen3-4B)Withheld: Not comparable: snapshots and subsets differ, so the ranking row shows no score.auROC or auPRC, varying by paper
Each paper builds its own ClinVar subset/snapshot
Self-reported
source second-hand Its own derived split; from an existing site note, not re-verified

Models reported

Grouped by relation type; each line shows its source type. Counted relations add up to 5. A pairing whose evidence rests only on this survey's own page is shown as unverified.

Developer report, modified protocol (5)

  • AlphaGenome Self-reported · Its own variant-effect suite, including splicing classes from this database †partly verified · evidence: link
  • AlphaMissense Self-reported · Its own missense test set · evidence: link
  • BioReason Self-reported · Its own pathway question task, not this protocol · evidence: link
  • Evo 2 Self-reported · Its own subsets, figure-only †partly verified · evidence: link
  • popEVE Self-reported · An expert-curated subset from earlier years · evidence: link

Caveats

  1. There is no official split, predictors and labels contaminate each other, and training on this database inflates scores.
  2. Pooled coding-missense auROC is already above 0.9, so gaps of 0.01 to 0.03 do not separate methods.
  3. The gene distribution is heavily skewed, with a few frequently tested genes dominating.
  4. The truth is submitted clinical interpretation rather than measurement, and submitters sometimes conflict.
  5. High auROC does not mean clinical usability, which requires calibrated evidence strength.

How to use it

Take a dated snapshot, filter by star level and variant class and score the held-out set; no wet lab of your own is needed. Report the snapshot date, star threshold, source of negatives and gene distribution, and state whether the model has seen these labels. Generalization claims need a time split or held-out genes plus calibrated evidence strength. Pair with ProteinGym and CAGI.

Pairs well with: ProteinGym · CAGI · TraitGym · Basenji/Enformer test set

Sources

  1. Official database site (official site)
  2. Database chapter of the official handbook (official site)
  3. Official release directory (repository)
  4. Documentation of review status stars (official site)
  5. Early description paper of the database (journal)
  6. Model paper reporting this protocol on its own subset (journal)

Corrections

3 corrections
FieldWasNowBasisFound
first_release_basisClinVar NAR database paper online (public launch reportedly Apr 2013, not verified)official full public release of ClinVar, April 2013 (NCBI Handbook); Nov 2012 prototype stored separately, not displayed as first release; exact April day not sourced.source2026-11-16
first_release2013-11-142013-04 (official launch; prototype Nov 2012)source2026-11-16
ranking.entriesper-model scores shown in the ranking rowscores withheld; each entry keeps its protocol description onlysource2026-11-16

Part of the AI for Bio benchmarks survey · all 44 benchmarks · open in the explorer. Record as of 2026-11-16.