Overview
| Name | CAGI · aliases: CAGI, CAGI5, CAGI6, CAGI7, Critical Assessment of Genome Interpretation |
|---|---|
| Full name | Critical Assessment of Genome Interpretation |
| Ambiguous names (not auto-linked) | None |
| Family | Not a family entry. Single recurring challenge; CAGI1–7 are editions of one challenge, not family members. |
| Domain | Genomics & RNA |
| Inclusion | Original 30 |
| Type | A · Recurring blind challenge with heterogeneous tasks per edition |
| Evidence level | L4–L5 · Annotate-All-Missense near L5 (post-deadline ClinVar) |
| Publisher | CAGI consortium (NIH-funded) (Academic lab) |
| First release | † partly verified Basis: first CAGI season organised in 2010 (year-only evidence). The 2010-12-10 Berkeley workshop is the closing event, not the opening; no organiser page states the opening month. Note: Year sourced; opening month has no primary source source |
| Other milestones | |
| Latest update | · new round Basis: Organiser website announcement Organisers announced the CAGI7 prediction season complete (14 challenges; RGP-CRAM opened 2025-12-27, deadlines extended to February 2026). source The most recent sourced event, not a claim that nothing changed since. |
| Next expected | The organiser site does not announce CAGI8. |
| Status | ● Active · Active; CAGI7 ran in 2025, conference December 2025 |
| Scale | CAGI7: 14 challenges (12 plus 2 off-cycle); CAGI1–5: 50 challenges, 738 submissions.
|
| Wet-lab refresh | Fresh experiments each round · Every 2–3 years, 12–14 challenges per edition; truth from data providers, no dedicated wet lab. New truth only by entering the next edition. |
| Own wet lab needed to evaluate a new model | No, run it as a download |
| Models reported | 1 counted (retrospective 1) |
| Unverified fields | † first_release, release_milestones.challenge_open, release_milestones.first_public_evidence, models_reported[0] |
| As of | 2026-11-16 |
What it measures
- Task: Blind assessment of variant interpretation across functional assays, splicing, rare-disease diagnosis and complex-trait prediction.
- Truth: Experimental (MAVE, enzyme activity) or clinical data unpublished at submission; Annotate-All uses ClinVar labels added after the deadline.
- Primary metric: Varies by challenge: Pearson/Kendall τ, AUC
- Secondary metrics: lr+, accuracy; Top-N ranking (rare-disease diagnosis)
- Version traps: No unified score across challenges; state edition, challenge and assessment-paper version (journal and preprint numbers may differ).
Current ranking
Third-party
| Metric | AUROC (pathogenic vs benign missense) |
|---|---|
| Eval set | CAGI6 Annotate-All-Missense full evaluation set (60 predictors) |
| Snapshot | stale |
| Board | https://genomeinterpretation.ccs.neu.edu/challenges.html |
| Retrieved | 2026-11-16 |
Caveat. From a third-party assessment paper (Hum Genet 144:281-293); AlphaMissense's overall rank is not stated (tie at 0.88); CAGI7 assessments unpublished; no unified CAGI board.
Shown in the overview table: Third-party
| Rank | Model | Score (verbatim) | Metric · eval set / protocol | Source type · date | Source and note |
|---|---|---|---|---|---|
| 1 | ClinPred / MetaRNN | AUROC 0.92 | AUROC (pathogenic vs benign missense) CAGI6 Annotate-All-Missense full evaluation set (60 predictors) | Third-party | source Not a site model; meta-predictors using allele frequency |
| — | VARITY_R | AUROC 0.88 | AUROC (pathogenic vs benign missense) CAGI6 Annotate-All-Missense full evaluation set (60 predictors) | Third-party | source Not a site model; meta-predictor without allele frequency |
| — | AlphaMissense | AUROC 0.88 | AUROC (pathogenic vs benign missense) CAGI6 Annotate-All-Missense full evaluation set (60 predictors) | Third-party | source Best individual (non-meta) predictor, tied with another; evaluated post hoc, not a CAGI submitter; no overall rank stated. |
Models reported
Grouped by relation type; each line shows its source type. Counted relations add up to 1. A pairing whose evidence rests only on this survey's own page is shown as unverified.
Retrospective dataset use (1)
- AlphaMissense Third-party · Post-hoc third-party evaluation on CAGI6 (checked at abstract level) †partly verified · evidence: link
Caveats
- Challenges are small and heterogeneous with no unified score; some have only tens of variants, assessors can clean data afterwards, and most predictors saw ClinVar or HGMD in training.
- AlphaMissense was the best individual predictor on CAGI6 (AUROC 0.88, third-party assessment, 2025), evaluated post hoc and behind two meta-predictors; not first of all 60.
- Absolute functional values are poorly calibrated and regulatory and complex-trait tasks remain weak; missense binary classification is near its empirical ceiling.
- No loop of selecting variants by prediction and then testing them, so no direct evidence of wet-lab translation.
How to use it
Enter the next edition, or test post hoc on released assessment data under a data-use agreement (no longer blind for later-cutoff models). No own wet lab needed; state edition, challenge and metric, and check training/evaluation overlap. Pair with ProteinGym (offline function) and ClinVar time hold-out (clinical).
Pairs well with: ProteinGym · ClinVar hold-out protocol · CAFA · CASP
Sources
- CAGI official site (official site)
- CAGI7 page (official site)
- CAGI challenge list (official site)
- CAGI summary paper, Genome Biology 25:53 (2024) (journal)
- CAGI6 Annotate-All-Missense assessment (Hum Genet 2025) (third-party paper)
- Same assessment, journal page (journal)
- Blog report on the first season (second-hand) (second-hand) second-hand
Corrections
4 corrections
| Field | Was | Now | Basis | Found |
|---|---|---|---|---|
| first_release | 2010 (fall); 2010-12-10 is the culminating Berkeley workshop, not the start | 2010 | source | 2026-11-16 |
| first_release_basis | Pre-CAGI/CAGI1 workshop in Berkeley 2010-12-10 (first round organized fall 2010) | first CAGI season organised in 2010 (year-only evidence). The 2010-12-10 Berkeley workshop is the closing event, not the opening; no organiser page states the opening month. | source | 2026-11-16 |
| first_release | 2010-12-10 | 2010 (fall); 2010-12-10 is the culminating Berkeley workshop, not the start | source | 2026-11-16 |
| ranking / models_reported (AlphaMissense) | AlphaMissense ranked first of 60 predictors in CAGI6 | Best individual (non-meta) predictor, AUROC 0.88, tied with another non-meta predictor, behind two meta-predictors (0.92); evaluated post hoc, not a CAGI submitter; no overall rank stated in the paper | source | 2026-11-16 |
Part of the AI for Bio benchmarks survey · all 44 benchmarks · open in the explorer. Record as of 2026-11-16.