AI for Bio · Benchmark 02

ProteinGym

Unified suite of 200+ public deep mutational scans for protein variant-effect and fitness ranking.

bench-02Type BEvidence L2–L3Proteins◐ Low-frequency Board中文

Overview

NameProteinGym · aliases: ProteinGym, ProteinGym v1, ProteinGym v1.3, PG_v1.3
Full nameThe name is not an abbreviation, so there is no expanded form.
Ambiguous names (not auto-linked)ProteinGym v0
Familyv0 is the substitution set inside the Tranception preprint; v1 onward is the standalone benchmark adding indels and a clinical subset. ViroGym and the R client are excluded.
  • ProteinGym v0 (inside the Tranception preprint) ·
  • ProteinGym v1 (standalone benchmark, NeurIPS 2023 Datasets and Benchmarks) ·
DomainProteins
InclusionOriginal 30
TypeB · Static DMS suite with a maintainer-run leaderboard
Evidence levelL2–L3 · Retrospective; supervised positional splits near L3
PublisherHarvard Medical School, Department of Systems Biology, University of Oxford, OATML group (Academic lab)
First release† partly verified · paper v1
Basis: Earliest family member: ProteinGym v0 in the Tranception preprint (arXiv v1); the standalone benchmark paper is bioRxiv 2023-12-08.
Note: Family-level date; the standalone benchmark first appeared 2023-12-08 source
Other milestones
  • First public evidence: · First appearance of the benchmark name, in the Tranception preprint source
  • Preprint v1: · Standalone benchmark, bioRxiv v1 source
  • Code release: † · PG_v1.3 data release on the repository releases page; the year comes from an existing site note, as the page shows only day and month source
Latest update† partly verified · data
Basis: Official repository commit history
A repository commit restored missing benchmark files; an earlier commit added a supervised clinical evaluation script. Neither carries a new release tag. source
The most recent sourced event, not a claim that nothing changed since.
Next expectedThe maintainers have not announced v1.4 or a next generation.
Status◐ Low-frequency · Active but low-frequency; latest data version v1.3 (2025)
Scalev1.3: 217 substitution assays, 74 indel assays, clinical subset of 2,525 proteins.
  • Substitution DMS assays: 217 (v1.3)
  • Substitution variants: about 2.7 million (v1.3)
  • Indel assays: 74 (v1.3)
  • Indel variants: about 300,000 (v1.3)
  • Proteins in the clinical subset; the data card also states 2,575: 2525 (v1.3)
  • Model rows on the official summary table, including sizes of one model: 97 (2026-11-16)
Wet-lab refreshGrows from public sources · No new experiments; published DMS assays are added irregularly and recent releases mainly added baselines.
Own wet lab needed to evaluate a new modelYes, to claim real generalization
Models reported10 counted (board 7, self 2, retrospective 1)
Unverified fields† first_release, release_milestones.code_release, latest_update, scale.items[4], models_reported[8], models_reported[9]
As of2026-11-16

What it measures

  • Task: Zero-shot and supervised ranking of protein fitness and variant effects over public deep mutational scans, plus a clinical subset.
  • Truth: Published DMS readouts (growth, expression, binding, catalysis, stability) and expert clinical labels for the clinical subset.
  • Primary metric: Average zero-shot Spearman
  • Secondary metrics: NDCG, AUC, MCC, Top-K recall; Supervised Spearman and MSE; AUC on the clinical subset
  • Version traps: State the data version and track (zero-shot, supervised, clinical): assay counts and baselines differ by version, so scores do not compare across versions.

Current ranking

Board

MetricAverage Spearman (zero-shot, DMS substitutions)
Eval setProteinGym v1.3, 217 substitution DMS assays, zero-shot
Snapshot stale
Boardhttps://proteingym.org/
Retrieved2026-11-16

Caveat. The summary table has 97 rows and gaps below about 0.01 fall within bootstrap standard error. Some models appear only in author or second-hand claims, not on the official table.

Shown in the overview table:

AIDO Protein-RAG (16B)
0.518 (SE 0.0)
Average Spearman (zero-shot, DMS substitutions) · ProteinGym v1.3, 217 substitution DMS assays, zero-shot
Board · stale
Best site model: VenusREM (#2)

RankModelScore (verbatim)Metric · eval set / protocolSource type · dateSource and note
1AIDO Protein-RAG (16B)0.518 (SE 0.0)Average Spearman (zero-shot, DMS substitutions)
ProteinGym v1.3, 217 substitution DMS assays, zero-shot
Board
source Not a site model; tied first; summary table last changed 2025-06-13, read 2026-11-16
2VenusREM0.518 (SE 0.005)Average Spearman (zero-shot, DMS substitutions)
ProteinGym v1.3, 217 substitution DMS assays, zero-shot
Board
source Ties first on score; the gap is within bootstrap standard error
3ProSST (K=2048)0.507 (SE 0.006)Average Spearman (zero-shot, DMS substitutions)
ProteinGym v1.3, 217 substitution DMS assays, zero-shot
Board
source Other K variants of this model rank 4th, 7th, 10th, 13th and 28th
14ESM3 open (1.4B)0.466 (SE 0.006)Average Spearman (zero-shot, DMS substitutions)
ProteinGym v1.3, 217 substitution DMS assays, zero-shot
Board
source
17SaProt (650M)0.457 (SE 0.007)Average Spearman (zero-shot, DMS substitutions)
ProteinGym v1.3, 217 substitution DMS assays, zero-shot
Board
source The 35M version of the same model ranks 48th
49ESM-C (300M)0.406 (SE 0.013)Average Spearman (zero-shot, DMS substitutions)
ProteinGym v1.3, 217 substitution DMS assays, zero-shot
Board
source The 600M version of the same model ranks 51st
54xTrimoPGLM-3B-MLM0.399 (SE 0.012)Average Spearman (zero-shot, DMS substitutions)
ProteinGym v1.3, 217 substitution DMS assays, zero-shot
Board
source Best-performing variant of this model; its 100B-int4 version ranks 74th
59Progen3 3B0.392 (SE 0.008)Average Spearman (zero-shot, DMS substitutions)
ProteinGym v1.3, 217 substitution DMS assays, zero-shot
Board
source The 1B version of the same model ranks 61st

Models reported

Grouped by relation type; each line shows its source type. Counted relations add up to 10. A pairing whose evidence rests only on this survey's own page is shown as unverified.

Official leaderboard or results (7)

  • ESM C Board DMS substitutions zero-shot, maintainer-run summary table (updated 2025-06-13); 300M and 600M entries · Entry on the official summary table · evidence: link
  • ESM3 Board DMS substitutions zero-shot, maintainer-run summary table (updated 2025-06-13); open 1.4B entry · Entry on the official summary table · evidence: link
  • ProGen3 Board · Entry on the official summary table · evidence: link
  • ProSST Board zero-shot substitution DMS summary table, several structure-vocabulary sizes · Entry on the official summary table · evidence: link
  • SaProt Board zero-shot substitution DMS summary · Entry on the official summary table · evidence: link
  • VenusREM Board zero-shot substitution official summary table (Structure & MSA category) · Entry on the official summary table · evidence: link
  • xTrimoPGLM Board · Entry on the official summary table · evidence: link

Developer report, official protocol (2)

  • PoET-2 Self-reported zero-shot (substitutions, indels, higher-order, clinical) plus supervised settings · Paper table claims the official protocol; not found on the official table · evidence: link
  • Profluent-E1 Self-reported · Only a curve in the project README; no row on the official summary table †partly verified · evidence: link

Retrospective dataset use (1)

  • Frontier general-purpose LLMs Third-party · Scores from a second-hand aggregator; the entry is a family, not one model low confidence †unverified · evidence: no source link

Mentioned, not a pairing

  • · Twelve of its DMS sets are used only as a data source; recorded explicitly as a non-pair so it is not re-added later. no source link

Caveats

  1. Static and tunable: mean Spearman hides assay heterogeneity and gaps below the bootstrap standard error (about 0.01) are not meaningful.
  2. No direct evidence links scores to wet-lab hit rates, and board position does not establish design or selection ability.
  3. Public data can be tuned against and there is no time freeze; the clinical subset risks circularity with predictors trained on the same clinical database.
  4. Metrics disagree: Spearman and NDCG pick different best models within assay categories.
  5. Coverage is skewed: most methods degrade systematically on viral proteins and proteins with disordered regions.

How to use it

Download the reference files, score every assay's variants, run the official scripts and submit by repository pull request; no own wet lab is needed. Report the data version, track, whether MSA or structure inputs were used, bootstrap standard errors and NDCG, split by assay type and taxon. Claims about design hit rates need a prospective assay of your own.

Pairs well with: ClinVar hold-out protocol · CAGI · FLIP / FLIP2 · RNAGym · TraitGym

Sources

  1. Official benchmark site (official site)
  2. Standalone benchmark paper on bioRxiv (NeurIPS 2023 Datasets and Benchmarks) (preprint)
  3. Tranception preprint, first appearance of the v0 substitution set (preprint)
  4. Official toolkit on PyPI (package registry)
  5. Third-party analysis of structure-input effects (third-party paper)

Corrections

3 corrections
FieldWasNowBasisFound
latest_update2026-01-122026-03-25source2026-11-16
first_releasestart year 20232022-05-27 at family level (v0 in the Tranception preprint), with the standalone benchmark at 2023-12-08 and the family boundary statedsource2026-11-16
n_models_reported910 under the new relation taxonomy, counting established relations onlysource2026-11-16

Part of the AI for Bio benchmarks survey · all 44 benchmarks · open in the explorer. Record as of 2026-11-16.