AI for Bio · Benchmark 17

CAFA

Time-delayed blind test of protein function (GO) prediction against experimental annotations added after submission.

bench-17Type AEvidence L4–L5Proteins● Active No ranking中文

Overview

NameCAFA · aliases: CAFA, CAFA1, CAFA2, CAFA3, CAFA4, CAFA5, CAFA6, Critical Assessment of Functional Annotation
Full nameCritical Assessment of Functional Annotation
Ambiguous names (not auto-linked)None
FamilyNot a family entry. Single recurring challenge; CAFA1–6 are editions of one challenge, not family members.
DomainProteins
InclusionOriginal 30
TypeA · Time-delayed recurring blind challenge
Evidence levelL4–L5 · CAFA3 added dedicated screens (L5)
PublisherCAFA organisers (ISCB Automated Function Prediction SIG) (Academic lab)
First release† partly verified
Basis: challenge_open: CAFA1 targets released 2010-09-15 (organisers' paper, p. 9). Registration opened 2010-07-15, stored separately and not used as first release.
Note: Day 2010-09-15 from the organisers' paper; kept at month source
Other milestones
  • First public evidence: † · Registration opened (second-hand quote of a 2010 announcement) source
  • Journal online: † · CAFA1 results paper online in Nature Methods (second-hand quote) source
  • Challenge opens: † · challenge_open: CAFA1 targets released 2010-09-15 (organisers' paper, p. 9). Registration opened 2010-07-15, stored separately and not used as first release. source
Latest update† unverified · new round
Basis: Organisers' mailing-list announcement
CAFA6 launched on Kaggle ($50k prizes) per the organisers' mailing-list post; final results not found.
Note: No official dated record; shown as unverified source
The most recent sourced event, not a claim that nothing changed since.
Next expectedCAFA6 final evaluation timing (about 2026-05) appears only on aggregator sites; nothing announced by the organisers.
Status● Active · CAFA6 opened on Kaggle 2025-10-16; no official results as of 2026-11-16
ScaleCAFA2: 126 methods, 3,681 proteins; CAFA5: 1,600+ teams; CAFA6 evaluation-set size not published.
  • Methods: 126 (CAFA2)
  • Proteins evaluated: 3,681 (CAFA2)
  • Teams: 1,600+ (CAFA5)
  • Submissions: 2,850 (CAFA5)
  • Evaluation-set size: not published (CAFA6)
Wet-lab refreshGrows from public sources · Every 2–3 years; new truth is experimental GO annotation accumulated in databases, not new experiments, and comes only by entering the next edition.
Own wet lab needed to evaluate a new modelNo, run it as a download
Models reported0 counted (unconfirmed 1)
Unverified fields† first_release, release_milestones.challenge_open, release_milestones.first_public_evidence, release_milestones.journal_online, latest_update, ranking.entries[0], models_reported[0]
As of2026-11-16

What it measures

  • Task: Time-delayed prediction of protein function as Gene Ontology terms, scored against experimental annotations added after submission.
  • Truth: Experimental-evidence GO annotations added to UniProt/GOA between t0 and t1, propagated up the GO hierarchy.
  • Primary metric: Fmax
  • Secondary metrics: Smin (IA-weighted); IA-weighted F-measure (CAFA5/6 Kaggle)
  • Version traps: CAFA1–4 used academic submissions with independent assessment, CAFA5/6 are Kaggle contests with different metrics, so scores are not comparable across editions; separate no-knowledge and limited-knowledge.

Current ranking

No ranking

MetricIA-weighted F-measure (CAFA-evaluator, Kaggle)
Eval setCAFA6 (Kaggle, launched 2025-10-16)
Snapshot
Boardhttps://www.kaggle.com/competitions/cafa-6-protein-function-prediction/leaderboard
Retrieved2026-11-16

Caveat. CAFA6 final board and winners not retrievable; no site model reports CAFA (SciReasoner pairing unconfirmed); CAFA5 scores are not comparable.

No headline entry: CAFA6 final board and winners were not found; the only entry is an interim snapshot, so there is no headline number.

RankModelScore (verbatim)Metric · eval set / protocolSource type · dateSource and note
1(individual Kaggle participant; interim)0.36IA-weighted F-measure (CAFA-evaluator, Kaggle)
CAFA6 (Kaggle, launched 2025-10-16)
Board
source Not a site model; interim public-board snapshot via a third-party aggregator with 57 days left, not final; participant name omitted.

Models reported

Grouped by relation type; each line shows its source type. Counted relations add up to 0. A pairing whose evidence rests only on this survey's own page is shown as unverified.

Unconfirmed (1, not counted)

  • SciReasoner Unconfirmed source · Self-reports on DeepFRI-GO; CAFA-3 only in a figure caption, a modified protocol if kept; paper identity unresolved (two same-name papers). †disputed · evidence: link

Caveats

  1. Unannotated terms count as negatives (open-world problem); CAFA5/6 moved to Kaggle, so scores are not comparable across editions; text-mining entries are allowed.
  2. Fmax favours shallow, frequent terms, and rankings shift substantially with the metric.
  3. Top CAFA5 teams used text mining of new papers as features, which is legal in time but changes what the task measures.
  4. Top-method gains from CAFA2 to CAFA3 were smaller than the previous step and CCO did not improve; the bottleneck is annotation, not models.
  5. CAFA3 included prospective experiments, but the mapping from experimental hit rate to Fmax is not quantified.

How to use it

Enter the next edition, or build a time split (train on an old snapshot, evaluate on a newer one) and compute Fmax and Smin with CAFA-evaluator, next to Naive and BLAST baselines, reporting no-knowledge and limited-knowledge and IA-weighted versions. No own wet lab needed, but scores reflect only documented function.

Pairs well with: CASP · CAGI · ProteinGym

Sources

  1. CAFA6 on Kaggle (official site)
  2. CAFA6 organisers' mailing-list announcement (official site)
  3. CAFA official site (official site)
  4. CAFA1 results paper (Nature Methods 2013) (journal)
  5. CAFA2 paper (PubMed) (journal)
  6. CAFA3 paper (Genome Biology 2019) (journal)
  7. Archived organisers' paper (CAFA1 dates) (archive)
  8. ProtBoost (CAFA5 team paper) (third-party paper)
  9. CAFA6 interim standings aggregator (second-hand) (second-hand) second-hand

Corrections

3 corrections
FieldWasNowBasisFound
first_release2010-09 (challenge began 2010-09-15; registration 2010-07-15)2010-09source2026-11-16
first_release_basisCAFA1 organized fall 2010 to spring 2012 per secondary summaries; month not foundchallenge_open: CAFA1 targets released 2010-09-15 (organisers' paper, p. 9). Registration opened 2010-07-15, stored separately and not used as first release.source2026-11-16
first_release2010-fall2010-09 (challenge began 2010-09-15; registration 2010-07-15)source2026-11-16

Part of the AI for Bio benchmarks survey · all 44 benchmarks · open in the explorer. Record as of 2026-11-16.