AI for Bio · Benchmark 13

CAGI

Long-running CASP-style blind test of variant interpretation, scored on data unpublished at submission.

bench-13Type AEvidence L4–L5Genomics & RNA● Active Third-party中文

Overview

NameCAGI · aliases: CAGI, CAGI5, CAGI6, CAGI7, Critical Assessment of Genome Interpretation
Full nameCritical Assessment of Genome Interpretation
Ambiguous names (not auto-linked)None
FamilyNot a family entry. Single recurring challenge; CAGI1–7 are editions of one challenge, not family members.
DomainGenomics & RNA
InclusionOriginal 30
TypeA · Recurring blind challenge with heterogeneous tasks per edition
Evidence levelL4–L5 · Annotate-All-Missense near L5 (post-deadline ClinVar)
PublisherCAGI consortium (NIH-funded) (Academic lab)
First release† partly verified
Basis: first CAGI season organised in 2010 (year-only evidence). The 2010-12-10 Berkeley workshop is the closing event, not the opening; no organiser page states the opening month.
Note: Year sourced; opening month has no primary source source
Other milestones
  • First public evidence: † · Blog report on the first season (2010-12) source
  • Challenge opens: † · first CAGI season organised in 2010 (year-only evidence). The 2010-12-10 Berkeley workshop is the closing event, not the opening; no organiser page states the opening month. source
Latest update · new round
Basis: Organiser website announcement
Organisers announced the CAGI7 prediction season complete (14 challenges; RGP-CRAM opened 2025-12-27, deadlines extended to February 2026). source
The most recent sourced event, not a claim that nothing changed since.
Next expectedThe organiser site does not announce CAGI8.
Status● Active · Active; CAGI7 ran in 2025, conference December 2025
ScaleCAGI7: 14 challenges (12 plus 2 off-cycle); CAGI1–5: 50 challenges, 738 submissions.
  • Challenges: 14 (CAGI7 (12 + 2 off-cycle))
  • Challenges: 13 + 1 (CAGI6)
  • Challenges: 50 (CAGI1–5)
  • Submissions: 738 (CAGI1–5)
  • Predictors assessed (Annotate-All-Missense): 60 (CAGI6)
Wet-lab refreshFresh experiments each round · Every 2–3 years, 12–14 challenges per edition; truth from data providers, no dedicated wet lab. New truth only by entering the next edition.
Own wet lab needed to evaluate a new modelNo, run it as a download
Models reported1 counted (retrospective 1)
Unverified fields† first_release, release_milestones.challenge_open, release_milestones.first_public_evidence, models_reported[0]
As of2026-11-16

What it measures

  • Task: Blind assessment of variant interpretation across functional assays, splicing, rare-disease diagnosis and complex-trait prediction.
  • Truth: Experimental (MAVE, enzyme activity) or clinical data unpublished at submission; Annotate-All uses ClinVar labels added after the deadline.
  • Primary metric: Varies by challenge: Pearson/Kendall τ, AUC
  • Secondary metrics: lr+, accuracy; Top-N ranking (rare-disease diagnosis)
  • Version traps: No unified score across challenges; state edition, challenge and assessment-paper version (journal and preprint numbers may differ).

Current ranking

Third-party

MetricAUROC (pathogenic vs benign missense)
Eval setCAGI6 Annotate-All-Missense full evaluation set (60 predictors)
Snapshot stale
Boardhttps://genomeinterpretation.ccs.neu.edu/challenges.html
Retrieved2026-11-16

Caveat. From a third-party assessment paper (Hum Genet 144:281-293); AlphaMissense's overall rank is not stated (tie at 0.88); CAGI7 assessments unpublished; no unified CAGI board.

Shown in the overview table:

ClinPred / MetaRNN
AUROC 0.92
AUROC (pathogenic vs benign missense) · CAGI6 Annotate-All-Missense full evaluation set (60 predictors)
Third-party · stale

RankModelScore (verbatim)Metric · eval set / protocolSource type · dateSource and note
1ClinPred / MetaRNNAUROC 0.92AUROC (pathogenic vs benign missense)
CAGI6 Annotate-All-Missense full evaluation set (60 predictors)
Third-party
source Not a site model; meta-predictors using allele frequency
—VARITY_RAUROC 0.88AUROC (pathogenic vs benign missense)
CAGI6 Annotate-All-Missense full evaluation set (60 predictors)
Third-party
source Not a site model; meta-predictor without allele frequency
—AlphaMissenseAUROC 0.88AUROC (pathogenic vs benign missense)
CAGI6 Annotate-All-Missense full evaluation set (60 predictors)
Third-party
source Best individual (non-meta) predictor, tied with another; evaluated post hoc, not a CAGI submitter; no overall rank stated.

Models reported

Grouped by relation type; each line shows its source type. Counted relations add up to 1. A pairing whose evidence rests only on this survey's own page is shown as unverified.

Retrospective dataset use (1)

  • AlphaMissense Third-party · Post-hoc third-party evaluation on CAGI6 (checked at abstract level) †partly verified · evidence: link

Caveats

  1. Challenges are small and heterogeneous with no unified score; some have only tens of variants, assessors can clean data afterwards, and most predictors saw ClinVar or HGMD in training.
  2. AlphaMissense was the best individual predictor on CAGI6 (AUROC 0.88, third-party assessment, 2025), evaluated post hoc and behind two meta-predictors; not first of all 60.
  3. Absolute functional values are poorly calibrated and regulatory and complex-trait tasks remain weak; missense binary classification is near its empirical ceiling.
  4. No loop of selecting variants by prediction and then testing them, so no direct evidence of wet-lab translation.

How to use it

Enter the next edition, or test post hoc on released assessment data under a data-use agreement (no longer blind for later-cutoff models). No own wet lab needed; state edition, challenge and metric, and check training/evaluation overlap. Pair with ProteinGym (offline function) and ClinVar time hold-out (clinical).

Pairs well with: ProteinGym · ClinVar hold-out protocol · CAFA · CASP

Sources

  1. CAGI official site (official site)
  2. CAGI7 page (official site)
  3. CAGI challenge list (official site)
  4. CAGI summary paper, Genome Biology 25:53 (2024) (journal)
  5. CAGI6 Annotate-All-Missense assessment (Hum Genet 2025) (third-party paper)
  6. Same assessment, journal page (journal)
  7. Blog report on the first season (second-hand) (second-hand) second-hand

Corrections

4 corrections
FieldWasNowBasisFound
first_release2010 (fall); 2010-12-10 is the culminating Berkeley workshop, not the start2010source2026-11-16
first_release_basisPre-CAGI/CAGI1 workshop in Berkeley 2010-12-10 (first round organized fall 2010)first CAGI season organised in 2010 (year-only evidence). The 2010-12-10 Berkeley workshop is the closing event, not the opening; no organiser page states the opening month.source2026-11-16
first_release2010-12-102010 (fall); 2010-12-10 is the culminating Berkeley workshop, not the startsource2026-11-16
ranking / models_reported (AlphaMissense)AlphaMissense ranked first of 60 predictors in CAGI6Best individual (non-meta) predictor, AUROC 0.88, tied with another non-meta predictor, behind two meta-predictors (0.92); evaluated post hoc, not a CAGI submitter; no overall rank stated in the papersource2026-11-16

Part of the AI for Bio benchmarks survey · all 44 benchmarks · open in the explorer. Record as of 2026-11-16.