AI for Bio · Benchmark 39

RNAGym

Zero-shot evaluation framework for RNA fitness and structure.

bench-39Type BEvidence L2Genomics & RNA● Active Board中文

Overview

NameRNAGym · aliases: RNAGym, RNAgym
Full nameThe authors use only the name RNAGym; no expanded form exists.
Ambiguous names (not auto-linked)None
FamilyNot a family entry. The three tasks belong to one evaluation framework, not a family.
DomainGenomics & RNA (also: Structure & co-folding)
InclusionSupplementary entry
TypeB · Fixed datasets, with the fitness part scored zero-shot.
Evidence levelL2 · Retrospective and zero-shot, with nothing blind or prospective.
PublisherHarvard Medical School (Academic lab)
First release · paper v1
Basis: bioRxiv preprint v1 (10.1101/2025.06.16.660049); the Hugging Face dataset creation date is not used as first-release evidence source
Other milestones
  • First public evidence: No sourced date† · The Hugging Face dataset was created 2025-05-15 and the workshop paper has no sourced posting date, so neither is used as first-release evidence. source
  • Preprint v1: · bioRxiv preprint v1 source
  • Code release: † · release v0.1.0 in the official repository (repository URL not recorded; see note)
Latest update† partly verified · board entry
Basis: commits to the leaderboard file in the official repository (fitness board update)
The fitness board gained two model entries and an outdated mRNA board was removed; a signed-Spearman metric arrived 2026-07-20, and v0.2 branches remain active. source
The most recent sourced event, not a claim that nothing changed since.
Next expectedThe v0.2 branches are active but the authors have announced no release date.
Status● Active · The fitness board was updated in 2026-08, the project website is noted as not yet synced, and the paper is still a preprint.
Scale70 DMS assays (over 1M mutations); 901k chemical-probing reactivity profiles; 215 PDB tertiary structures.
  • DMS assays: 70 (paper v1)
  • Variants: 1,117,295 (third-party tabulation)
  • Chemical-probing profiles: 901,000 (paper v1)
  • Tertiary structures: 215 (paper v1)
  • Assays on the fitness board: 31 (board, ncRNA only)
Wet-lab refreshStatic data · Reuses published DMS and chemical-probing experiments; nothing is added.
Own wet lab needed to evaluate a new modelNo, run it as a download
Models reported1 counted (self 1, unconfirmed 1)
Unverified fields† release_milestones.first_public_evidence, release_milestones.code_release, latest_update, models_reported[1], scale.items[1], ranking.entries, ranking.board_url
As of2026-11-16

What it measures

  • Task: Three tasks in one framework: zero-shot RNA fitness scoring, secondary-structure reactivity prediction and tertiary-structure prediction.
  • Truth: Published DMS fitness measurements, chemical-probing reactivity profiles and PDB structures.
  • Primary metric: Spearman correlation against measured fitness.
  • Secondary metrics: AUROC and MCC.; Separate metrics for the structure tasks.
  • Version traps: From v0.1.1 the metric is signed Spearman over ncRNA only, so earlier numbers are not comparable; aggregate within RNA category before equal-weight averaging and state whether mRNA assays are included.

Current ranking

Board

MetricSigned Spearman correlation, macro-averaged over three ncRNA categories with equal category weight (from v0.1.1)
Eval setRNAGym fitness board, ncRNA only (31 assays); mRNA coding and splicing assays excluded
Snapshot† partly verified
Boardhttps://rnagym.org
Retrieved2026-11-16

Caveat. Values are read from the leaderboard file in the official repository, which the project site had not yet synced; the v0.1.1 metric change breaks comparability with earlier figures; the repository warns the macro-mean over-weights two tiny categories and gaps under about 0.01 are meaningless; only fitness has a board.

Shown in the overview table:

0.2120 (Ribozyme 0.1081 / tRNA 0.4310 / Aptamer 0.0970)
Signed Spearman correlation, macro-averaged over three ncRNA categories with equal category weight (from v0.1.1) · RNAGym fitness leaderboard, ncRNA only (31 assays); mRNA coding/splicing assays excluded
Board · †

RankModelScore (verbatim)Metric · eval set / protocolSource type · dateSource and note
1Evo 2 (40B)0.2120 (Ribozyme 0.1081 / tRNA 0.4310 / Aptamer 0.0970)Signed Spearman correlation, macro-averaged over three ncRNA categories with equal category weight (from v0.1.1)
RNAGym fitness leaderboard, ncRNA only (31 assays); mRNA coding/splicing assays excluded
Board
†
source First overall on the board and a site model.
2RNA-ERNIE0.1937Signed Spearman correlation, macro-averaged over three ncRNA categories with equal category weight (from v0.1.1)
RNAGym fitness leaderboard, ncRNA only (31 assays); mRNA coding/splicing assays excluded
Board
†
source Not a site model.
3Evo 2 (7B)0.1904 (Ribozyme 0.0651 / tRNA 0.3867 / Aptamer 0.1192)Signed Spearman correlation, macro-averaged over three ncRNA categories with equal category weight (from v0.1.1)
RNAGym fitness leaderboard, ncRNA only (31 assays); mRNA coding/splicing assays excluded
Board
†
source A second checkpoint of the same model; two earlier generations rank seventh and eleventh.
—RIBOSPAN (1K-40 checkpoint)Spearman 0.2524, AUROC 0.6280, MCC 0.1904Spearman, AUROC and MCC under the developer's own aggregation
RNAGym fitness assays under the developer paper's own aggregation (not the current board protocol)
Self-reported
source Not on the official board; developer-reported under the paper's own aggregation with a different metric and assay scope, so no rank is given.

Models reported

Grouped by relation type; each line shows its source type. Counted relations add up to 1. A pairing whose evidence rests only on this survey's own page is shown as unverified.

Developer report, official protocol (1)

  • RIBOSPAN Self-reported · The developer self-reports zero-shot results on this set's fitness task under its own aggregation. · evidence: link

Unconfirmed (1, not counted)

  • Evo 2 Unconfirmed source · The site matrix records a board entry while this page's figure is quoted via a third-party paper; the board entry awaits re-checking. †disputed · evidence: link

Caveats

  1. The fitness part is zero-shot only, which an independent team argues is too narrow for fitness prediction.
  2. Data come from published DMS studies that may overlap model pretraining corpora, with no deduplication audit.
  3. Only fitness has a board; the structure tasks have baseline scripts but no board, and the board covers ncRNA only.
  4. Under the macro-mean, the tRNA and aptamer categories hold only single-digit assay counts yet carry full weight.
  5. Naming and protocol follow ProteinGym, but a larger nucleic-acid suite has overtaken its coverage and counts data differently.

How to use it

Download the data and baseline scripts and score each assay zero-shot: log-likelihood differences for autoregressive models, masked-marginals for masked ones. Aggregate within RNA category before equal-weight averaging rather than averaging over all assays; report Spearman, AUROC and MCC, and state the sequence-window handling, whether mRNA assays are included and the metric version. Structure tasks need their own pipeline, and a larger nucleic-acid fitness suite makes a useful scale comparison.

Pairs well with: ProteinGym · RNA-Puzzles · FLIP / FLIP2 · GUE + Nucleotide Transformer benchmark

Sources

  1. RNAGym preprint (bioRxiv 10.1101/2025.06.16.660049) (preprint) ·
  2. Project site rnagym.org, which links the official repository (official site)
  3. Fitness leaderboard file in the official repository, reached via the project site (leaderboard) · †
  4. ICLR 2025 workshop entry (official site)
  5. Third-party use: the RIBOSPAN paper (third-party paper)
  6. Third-party comparison: the NABench paper (third-party paper)

Corrections

3 corrections
FieldWasNowBasisFound
first_release2025-05-152025-06-17source2026-11-16
first_release_basisHugging Face dataset created (public data release); ICLR 2025 workshop paper same spring; bioRxiv v1 2025-06-17bioRxiv preprint v1, 2025-06-17; the dataset creation date and the undated workshop paper are notes onlysource2026-11-16
status_noteno official leaderboard page founda fitness leaderboard is published in the repository and was updated 2026-08-14source2026-11-16

Part of the AI for Bio benchmarks survey · all 44 benchmarks · open in the explorer. Record as of 2026-11-16.