AI for Bio · Benchmark 15

CAMEO

Weekly rolling blind test of structure-prediction servers on PDB pre-release sequences.

bench-15Type AEvidence L4Structure & co-folding● Active Third-party中文

Overview

NameCAMEO · aliases: CAMEO, CAMEO3D, cameo3d.org
Full nameContinuous Automated Model EvaluatiOn
Ambiguous names (not auto-linked)CAMEO 2022
FamilyNot a family entry. Single rolling evaluation service; static CAMEO subsets taken by papers are not family members.
DomainStructure & co-folding
InclusionOriginal 30
TypeA · Weekly automated rolling blind test, no human review
Evidence levelL4
PublisherSIB Swiss Institute of Bioinformatics, University of Basel, Biozentrum (Academic lab)
First release† partly verified
Basis: Maintainer papers state continuous weekly benchmarking since 2012; month of first targets not found
Note: Year sourced; archive week count implies early 2012 only by arithmetic source
Other milestones
  • First public evidence: † · Maintainer papers state continuous weekly benchmarking since 2012; month of first targets not found source
Latest updateNo sourced date† · no later event
Basis: No sourced event: cameo3d.org has no news or changelog
No later sourced event found (as of 2026-11-16): cameo3d.org has no news or changelog and the weekly archive gives no dated release.
Note: 2026-10-17 is only a Wayback snapshot date, not an update source
The most recent sourced event, not a claim that nothing changed since.
Next expectedRuns weekly on a rolling basis; no separately announced next event.
Status● Active · Active but reduced; single-chain 3D track ended April 2025 (unverified)
ScaleJan 2024–Jun 2025: 12,205 targets over 76 weeks; about 160 targets per week.
  • New PDB entries screened: 23,629 (2024-01-06 to 2025-06-14)
  • Targets selected: 12,205 (2024-01-06 to 2025-06-14)
  • Targets medium / hard / ligand: 2,106 / 3,266 / 6,833 (2024-01-06 to 2025-06-14)
  • Weeks: 76 (2024-01-06 to 2025-06-14)
  • Targets per week (site arithmetic): ~160
Wet-lab refreshFresh experiments each round · Weekly; truth is newly released PDB structures, about 160 per week, not produced for the benchmark.
Own wet lab needed to evaluate a new modelNo, run it as a download
Models reported4 counted (board 1, retrospective 3, unconfirmed 1)
Unverified fields† first_release, release_milestones.first_public_evidence, latest_update, status_note, ranking.entries[1], models_reported[0], models_reported[2]
As of2026-11-16

What it measures

  • Task: Weekly blind evaluation of structure-prediction servers on PDB pre-release sequences; now mainly complex and ligand tracks.
  • Truth: PDB structures released afterwards (X-ray, NMR, EM; resolution ≤4.0 Å).
  • Primary metric: lDDT family (iLDDT, oligo-lDDT)
  • Secondary metrics: QS-score; DockQ; lDDT-PLI (ligands)
  • Version traps: Papers use different windows and filters, so state the window; a static 'CAMEO 2022' subset is not the live service.

Current ranking

Third-party

MetriclDDT / ligand success rate (CAMEO complex and ligand tracks)
Eval setMixed windows: CAMEO 2024-01-06 to 2025-06-14 (AF3 baseline run by CAMEO); CAMEO 2022-04 to 2022-06 static set (xTrimoPGLM)
Snapshot stale
Boardhttps://cameo3d.org/
Retrieved2026-11-16

Caveat. The live site renders server results dynamically, with no retrievable text board; single-chain 3D closed April 2025 (site note); entries use different windows and are not comparable.

No headline entry: No headline number: the ranking source has no rank-1 entry (entries are unranked or protocol-specific); the caveat text is shown instead.

RankModelScore (verbatim)Metric · eval set / protocolSource type · dateSource and note
—AlphaFold 3 (CAMEO baseline)drug-like ligand success rate ~40%; antibody median LDDT 0.81lDDT / ligand success rate (CAMEO complex and ligand tracks)
CAMEO 2024-01-06 to 2025-06-14 (76 weeks), AF3 v3.0.1 run by CAMEO as baseline
Third-party
source CAMEO assessment paper (Proteins 2025); best baseline in most ligand categories; behind stoichiometry-aware servers on oligomers; window end date inconsistent in paper.
—xTrimoPGLM-FoldTM-score 0.86 / GDT_TS 0.83 (ESMFold 0.85/0.81; AF2 0.87/0.85)TM-score / GDT_TS
CAMEO 2022-04 to 2022-06 static set (n=194)
Self-reported
source second-hand Developer report on a static subset; from site note, not re-verified.
—Profluent-E1No scoreLong-range contact precision
CAMEO targets (window not stated in README)
Self-reported
source Shown only as a figure; no numeric score.

Models reported

Grouped by relation type; each line shows its source type. Counted relations add up to 4. A pairing whose evidence rests only on this survey's own page is shown as unverified.

Official leaderboard or results (1)

  • AlphaFold 3 Board operator-run complex and ligand baseline, 2024-01 to 2025-06 · CAMEO operators ran AF3 as a baseline (2025 assessment paper); maintainer-run, not a developer submission. †partly verified · evidence: link

Retrospective dataset use (3)

  • DPLM-2 Third-party frozen 2022 target set, folding and inverse folding · Uses a frozen 2022 target set; no live submission, no service ranking. Resolution: The paper uses a frozen 2022 target set derived from the service, with no live submission and no service-assigned ranking. Recorded as a relation rather than deleted, so the rule matches Nesso-1 x CASP16. · evidence: link
  • Profluent-E1 Third-party · CAMEO targets used post hoc; curve only. †partly verified · evidence: link
  • xTrimoPGLM Third-party · CAMEO 2022-04 to 06 static set, n=194. · evidence: link

Unconfirmed (1, not counted)

  • Enchant v1/v2 Unconfirmed source · Same source; static 2022 set, figure only. Unconfirmed; would count as retrospective use, not participation. low confidence †disputed · evidence: link

Caveats

  1. Windows differ between papers, so numbers are not comparable; difficulty tiers drift as servers improve, no confidence intervals are given, and new tools are hard to enter as servers.
  2. Pre-release data carry no stoichiometry, so homo-oligomers and monomers cannot be told apart, which affects complex scoring.
  3. Tools such as AlphaFold 3 have usage limits that hinder automated submission (second-hand report), so CAMEO runs it itself as a baseline.
  4. Truth is structure, not function or affinity; no direct evidence links lDDT gains to drug-discovery hit rates.

How to use it

No wet lab needed. Register as a server (receive targets automatically, return models on time), or replay contemporaneous new PDB structures offline and score with OpenStructure; the training cutoff must precede the window, and the window and filters must be stated. Not a basis for a single SOTA claim; pair with CASP, PoseBusters and PLINDER.

Pairs well with: CASP · PoseBusters · PLINDER / PINDER / Runs N' Poses · FoldBench

Sources

  1. CAMEO official site (official site)
  2. CAMEO assessment paper, Proteins 2025 (beyond single chains) (journal)
  3. CAMEO maintainer paper (Proteins) (journal)
  4. CAMEO 2018 paper (PubMed) (journal)
  5. xTrimoPGLM paper (developer) (preprint)
  6. cameo3d.org archive snapshot (2026-10-17) (archive)

Corrections

5 corrections
FieldWasNowBasisFound
first_release20122012source2026-11-16
latest_update2026-10-17nullsource2026-11-16
latest_update_descWayback snapshot of cameo3d.org shows weekly benchmarking live; no dated news/changes retrievable, last substantive change undatable.No dated update found: cameo3d.org carries no news or changelog and the weekly target archive gives no dated release. Must display as unverified.source2026-11-16
latest_update_kindcode-maintenance-onlynone-since-releasesource2026-11-16
models_reported (DPLM-2)No explained relation on the siteretrospective-dataset-usesource2026-11-16

Part of the AI for Bio benchmarks survey · all 44 benchmarks · open in the explorer. Record as of 2026-11-16.