Overview
| Name | ProteinGym · aliases: ProteinGym, ProteinGym v1, ProteinGym v1.3, PG_v1.3 |
|---|---|
| Full name | The name is not an abbreviation, so there is no expanded form. |
| Ambiguous names (not auto-linked) | ProteinGym v0 |
| Family | v0 is the substitution set inside the Tranception preprint; v1 onward is the standalone benchmark adding indels and a clinical subset. ViroGym and the R client are excluded.
|
| Domain | Proteins |
| Inclusion | Original 30 |
| Type | B · Static DMS suite with a maintainer-run leaderboard |
| Evidence level | L2–L3 · Retrospective; supervised positional splits near L3 |
| Publisher | Harvard Medical School, Department of Systems Biology, University of Oxford, OATML group (Academic lab) |
| First release | † partly verified · paper v1 Basis: Earliest family member: ProteinGym v0 in the Tranception preprint (arXiv v1); the standalone benchmark paper is bioRxiv 2023-12-08. Note: Family-level date; the standalone benchmark first appeared 2023-12-08 source |
| Other milestones |
|
| Latest update | † partly verified · data Basis: Official repository commit history A repository commit restored missing benchmark files; an earlier commit added a supervised clinical evaluation script. Neither carries a new release tag. source The most recent sourced event, not a claim that nothing changed since. |
| Next expected | The maintainers have not announced v1.4 or a next generation. |
| Status | ◐ Low-frequency · Active but low-frequency; latest data version v1.3 (2025) |
| Scale | v1.3: 217 substitution assays, 74 indel assays, clinical subset of 2,525 proteins.
|
| Wet-lab refresh | Grows from public sources · No new experiments; published DMS assays are added irregularly and recent releases mainly added baselines. |
| Own wet lab needed to evaluate a new model | Yes, to claim real generalization |
| Models reported | 10 counted (board 7, self 2, retrospective 1) |
| Unverified fields | † first_release, release_milestones.code_release, latest_update, scale.items[4], models_reported[8], models_reported[9] |
| As of | 2026-11-16 |
What it measures
- Task: Zero-shot and supervised ranking of protein fitness and variant effects over public deep mutational scans, plus a clinical subset.
- Truth: Published DMS readouts (growth, expression, binding, catalysis, stability) and expert clinical labels for the clinical subset.
- Primary metric: Average zero-shot Spearman
- Secondary metrics: NDCG, AUC, MCC, Top-K recall; Supervised Spearman and MSE; AUC on the clinical subset
- Version traps: State the data version and track (zero-shot, supervised, clinical): assay counts and baselines differ by version, so scores do not compare across versions.
Current ranking
Board
| Metric | Average Spearman (zero-shot, DMS substitutions) |
|---|---|
| Eval set | ProteinGym v1.3, 217 substitution DMS assays, zero-shot |
| Snapshot | stale |
| Board | https://proteingym.org/ |
| Retrieved | 2026-11-16 |
Caveat. The summary table has 97 rows and gaps below about 0.01 fall within bootstrap standard error. Some models appear only in author or second-hand claims, not on the official table.
Shown in the overview table: Board
| Rank | Model | Score (verbatim) | Metric · eval set / protocol | Source type · date | Source and note |
|---|---|---|---|---|---|
| 1 | AIDO Protein-RAG (16B) | 0.518 (SE 0.0) | Average Spearman (zero-shot, DMS substitutions) ProteinGym v1.3, 217 substitution DMS assays, zero-shot | Board | source Not a site model; tied first; summary table last changed 2025-06-13, read 2026-11-16 |
| 2 | VenusREM | 0.518 (SE 0.005) | Average Spearman (zero-shot, DMS substitutions) ProteinGym v1.3, 217 substitution DMS assays, zero-shot | Board | source Ties first on score; the gap is within bootstrap standard error |
| 3 | ProSST (K=2048) | 0.507 (SE 0.006) | Average Spearman (zero-shot, DMS substitutions) ProteinGym v1.3, 217 substitution DMS assays, zero-shot | Board | source Other K variants of this model rank 4th, 7th, 10th, 13th and 28th |
| 14 | ESM3 open (1.4B) | 0.466 (SE 0.006) | Average Spearman (zero-shot, DMS substitutions) ProteinGym v1.3, 217 substitution DMS assays, zero-shot | Board | source |
| 17 | SaProt (650M) | 0.457 (SE 0.007) | Average Spearman (zero-shot, DMS substitutions) ProteinGym v1.3, 217 substitution DMS assays, zero-shot | Board | source The 35M version of the same model ranks 48th |
| 49 | ESM-C (300M) | 0.406 (SE 0.013) | Average Spearman (zero-shot, DMS substitutions) ProteinGym v1.3, 217 substitution DMS assays, zero-shot | Board | source The 600M version of the same model ranks 51st |
| 54 | xTrimoPGLM-3B-MLM | 0.399 (SE 0.012) | Average Spearman (zero-shot, DMS substitutions) ProteinGym v1.3, 217 substitution DMS assays, zero-shot | Board | source Best-performing variant of this model; its 100B-int4 version ranks 74th |
| 59 | Progen3 3B | 0.392 (SE 0.008) | Average Spearman (zero-shot, DMS substitutions) ProteinGym v1.3, 217 substitution DMS assays, zero-shot | Board | source The 1B version of the same model ranks 61st |
Models reported
Grouped by relation type; each line shows its source type. Counted relations add up to 10. A pairing whose evidence rests only on this survey's own page is shown as unverified.
Official leaderboard or results (7)
- ESM C Board DMS substitutions zero-shot, maintainer-run summary table (updated 2025-06-13); 300M and 600M entries · Entry on the official summary table · evidence: link
- ESM3 Board DMS substitutions zero-shot, maintainer-run summary table (updated 2025-06-13); open 1.4B entry · Entry on the official summary table · evidence: link
- ProGen3 Board · Entry on the official summary table · evidence: link
- ProSST Board zero-shot substitution DMS summary table, several structure-vocabulary sizes · Entry on the official summary table · evidence: link
- SaProt Board zero-shot substitution DMS summary · Entry on the official summary table · evidence: link
- VenusREM Board zero-shot substitution official summary table (Structure & MSA category) · Entry on the official summary table · evidence: link
- xTrimoPGLM Board · Entry on the official summary table · evidence: link
Developer report, official protocol (2)
- PoET-2 Self-reported zero-shot (substitutions, indels, higher-order, clinical) plus supervised settings · Paper table claims the official protocol; not found on the official table · evidence: link
- Profluent-E1 Self-reported · Only a curve in the project README; no row on the official summary table †partly verified · evidence: link
Retrospective dataset use (1)
- Frontier general-purpose LLMs Third-party · Scores from a second-hand aggregator; the entry is a family, not one model low confidence †unverified · evidence: no source link
Mentioned, not a pairing
- · Twelve of its DMS sets are used only as a data source; recorded explicitly as a non-pair so it is not re-added later. no source link
Caveats
- Static and tunable: mean Spearman hides assay heterogeneity and gaps below the bootstrap standard error (about 0.01) are not meaningful.
- No direct evidence links scores to wet-lab hit rates, and board position does not establish design or selection ability.
- Public data can be tuned against and there is no time freeze; the clinical subset risks circularity with predictors trained on the same clinical database.
- Metrics disagree: Spearman and NDCG pick different best models within assay categories.
- Coverage is skewed: most methods degrade systematically on viral proteins and proteins with disordered regions.
How to use it
Download the reference files, score every assay's variants, run the official scripts and submit by repository pull request; no own wet lab is needed. Report the data version, track, whether MSA or structure inputs were used, bootstrap standard errors and NDCG, split by assay type and taxon. Claims about design hit rates need a prospective assay of your own.
Pairs well with: ClinVar hold-out protocol · CAGI · FLIP / FLIP2 · RNAGym · TraitGym
Sources
- Official benchmark site (official site)
- Standalone benchmark paper on bioRxiv (NeurIPS 2023 Datasets and Benchmarks) (preprint)
- Tranception preprint, first appearance of the v0 substitution set (preprint)
- Official toolkit on PyPI (package registry)
- Third-party analysis of structure-input effects (third-party paper)
Corrections
3 corrections
| Field | Was | Now | Basis | Found |
|---|---|---|---|---|
| latest_update | 2026-01-12 | 2026-03-25 | source | 2026-11-16 |
| first_release | start year 2023 | 2022-05-27 at family level (v0 in the Tranception preprint), with the standalone benchmark at 2023-12-08 and the family boundary stated | source | 2026-11-16 |
| n_models_reported | 9 | 10 under the new relation taxonomy, counting established relations only | source | 2026-11-16 |
Part of the AI for Bio benchmarks survey · all 44 benchmarks · open in the explorer. Record as of 2026-11-16.