Overview
| Name | RNAGym · aliases: RNAGym, RNAgym |
|---|---|
| Full name | The authors use only the name RNAGym; no expanded form exists. |
| Ambiguous names (not auto-linked) | None |
| Family | Not a family entry. The three tasks belong to one evaluation framework, not a family. |
| Domain | Genomics & RNA (also: Structure & co-folding) |
| Inclusion | Supplementary entry |
| Type | B · Fixed datasets, with the fitness part scored zero-shot. |
| Evidence level | L2 · Retrospective and zero-shot, with nothing blind or prospective. |
| Publisher | Harvard Medical School (Academic lab) |
| First release | · paper v1 Basis: bioRxiv preprint v1 (10.1101/2025.06.16.660049); the Hugging Face dataset creation date is not used as first-release evidence source |
| Other milestones |
|
| Latest update | † partly verified · board entry Basis: commits to the leaderboard file in the official repository (fitness board update) The fitness board gained two model entries and an outdated mRNA board was removed; a signed-Spearman metric arrived 2026-07-20, and v0.2 branches remain active. source The most recent sourced event, not a claim that nothing changed since. |
| Next expected | The v0.2 branches are active but the authors have announced no release date. |
| Status | ● Active · The fitness board was updated in 2026-08, the project website is noted as not yet synced, and the paper is still a preprint. |
| Scale | 70 DMS assays (over 1M mutations); 901k chemical-probing reactivity profiles; 215 PDB tertiary structures.
|
| Wet-lab refresh | Static data · Reuses published DMS and chemical-probing experiments; nothing is added. |
| Own wet lab needed to evaluate a new model | No, run it as a download |
| Models reported | 1 counted (self 1, unconfirmed 1) |
| Unverified fields | † release_milestones.first_public_evidence, release_milestones.code_release, latest_update, models_reported[1], scale.items[1], ranking.entries, ranking.board_url |
| As of | 2026-11-16 |
What it measures
- Task: Three tasks in one framework: zero-shot RNA fitness scoring, secondary-structure reactivity prediction and tertiary-structure prediction.
- Truth: Published DMS fitness measurements, chemical-probing reactivity profiles and PDB structures.
- Primary metric: Spearman correlation against measured fitness.
- Secondary metrics: AUROC and MCC.; Separate metrics for the structure tasks.
- Version traps: From v0.1.1 the metric is signed Spearman over ncRNA only, so earlier numbers are not comparable; aggregate within RNA category before equal-weight averaging and state whether mRNA assays are included.
Current ranking
Board
| Metric | Signed Spearman correlation, macro-averaged over three ncRNA categories with equal category weight (from v0.1.1) |
|---|---|
| Eval set | RNAGym fitness board, ncRNA only (31 assays); mRNA coding and splicing assays excluded |
| Snapshot | † partly verified |
| Board | https://rnagym.org |
| Retrieved | 2026-11-16 |
Caveat. Values are read from the leaderboard file in the official repository, which the project site had not yet synced; the v0.1.1 metric change breaks comparability with earlier figures; the repository warns the macro-mean over-weights two tiny categories and gaps under about 0.01 are meaningless; only fitness has a board.
Shown in the overview table: Board
| Rank | Model | Score (verbatim) | Metric · eval set / protocol | Source type · date | Source and note |
|---|---|---|---|---|---|
| 1 | Evo 2 (40B) | 0.2120 (Ribozyme 0.1081 / tRNA 0.4310 / Aptamer 0.0970) | Signed Spearman correlation, macro-averaged over three ncRNA categories with equal category weight (from v0.1.1) RNAGym fitness leaderboard, ncRNA only (31 assays); mRNA coding/splicing assays excluded | Board † | source First overall on the board and a site model. |
| 2 | RNA-ERNIE | 0.1937 | Signed Spearman correlation, macro-averaged over three ncRNA categories with equal category weight (from v0.1.1) RNAGym fitness leaderboard, ncRNA only (31 assays); mRNA coding/splicing assays excluded | Board † | source Not a site model. |
| 3 | Evo 2 (7B) | 0.1904 (Ribozyme 0.0651 / tRNA 0.3867 / Aptamer 0.1192) | Signed Spearman correlation, macro-averaged over three ncRNA categories with equal category weight (from v0.1.1) RNAGym fitness leaderboard, ncRNA only (31 assays); mRNA coding/splicing assays excluded | Board † | source A second checkpoint of the same model; two earlier generations rank seventh and eleventh. |
| — | RIBOSPAN (1K-40 checkpoint) | Spearman 0.2524, AUROC 0.6280, MCC 0.1904 | Spearman, AUROC and MCC under the developer's own aggregation RNAGym fitness assays under the developer paper's own aggregation (not the current board protocol) | Self-reported | source Not on the official board; developer-reported under the paper's own aggregation with a different metric and assay scope, so no rank is given. |
Models reported
Grouped by relation type; each line shows its source type. Counted relations add up to 1. A pairing whose evidence rests only on this survey's own page is shown as unverified.
Developer report, official protocol (1)
- RIBOSPAN Self-reported · The developer self-reports zero-shot results on this set's fitness task under its own aggregation. · evidence: link
Unconfirmed (1, not counted)
- Evo 2 Unconfirmed source · The site matrix records a board entry while this page's figure is quoted via a third-party paper; the board entry awaits re-checking. †disputed · evidence: link
Caveats
- The fitness part is zero-shot only, which an independent team argues is too narrow for fitness prediction.
- Data come from published DMS studies that may overlap model pretraining corpora, with no deduplication audit.
- Only fitness has a board; the structure tasks have baseline scripts but no board, and the board covers ncRNA only.
- Under the macro-mean, the tRNA and aptamer categories hold only single-digit assay counts yet carry full weight.
- Naming and protocol follow ProteinGym, but a larger nucleic-acid suite has overtaken its coverage and counts data differently.
How to use it
Download the data and baseline scripts and score each assay zero-shot: log-likelihood differences for autoregressive models, masked-marginals for masked ones. Aggregate within RNA category before equal-weight averaging rather than averaging over all assays; report Spearman, AUROC and MCC, and state the sequence-window handling, whether mRNA assays are included and the metric version. Structure tasks need their own pipeline, and a larger nucleic-acid fitness suite makes a useful scale comparison.
Pairs well with: ProteinGym · RNA-Puzzles · FLIP / FLIP2 · GUE + Nucleotide Transformer benchmark
Sources
- RNAGym preprint (bioRxiv 10.1101/2025.06.16.660049) (preprint) ·
- Project site rnagym.org, which links the official repository (official site)
- Fitness leaderboard file in the official repository, reached via the project site (leaderboard) · †
- ICLR 2025 workshop entry (official site)
- Third-party use: the RIBOSPAN paper (third-party paper)
- Third-party comparison: the NABench paper (third-party paper)
Corrections
3 corrections
| Field | Was | Now | Basis | Found |
|---|---|---|---|---|
| first_release | 2025-05-15 | 2025-06-17 | source | 2026-11-16 |
| first_release_basis | Hugging Face dataset created (public data release); ICLR 2025 workshop paper same spring; bioRxiv v1 2025-06-17 | bioRxiv preprint v1, 2025-06-17; the dataset creation date and the undated workshop paper are notes only | source | 2026-11-16 |
| status_note | no official leaderboard page found | a fitness leaderboard is published in the repository and was updated 2026-08-14 | source | 2026-11-16 |
Part of the AI for Bio benchmarks survey · all 44 benchmarks · open in the explorer. Record as of 2026-11-16.