AI for Bio · Benchmark 40

MotifBench

Thirty fixed motif-scaffolding problems judged by one pipeline, with a submission leaderboard; ground truth stays computational.

bench-40Type DEvidence L2Proteins● Active Board中文

Overview

NameMotifBench · aliases: MotifBench
Full nameThe name is already the full name; no expanded form is given by the sources.
Ambiguous names (not auto-linked)None
FamilyNot a family entry. One fixed problem set and one leaderboard; no family members.
DomainProteins (also: Molecule generation)
InclusionSupplementary entry
TypeD · Computational-oracle problem set with a submission leaderboard
Evidence levelL2 · Computational self-consistency: ProteinMPNN redesign plus ESMFold refolding
PublisherMulti-lab academic collaboration (MotifBench maintainers) (Academic lab)
First release · paper v1
Basis: arXiv 2502.12479 v1 (repository created 2024-06-14, not used as first release) source
Other milestones
  • First public evidence: · Earliest sourced public evidence is paper v1; the 2024-06-14 repository creation date is a note only, never a first release source
  • Preprint v1: · arXiv 2502.12479 v1 source
Latest update · board entry
Basis: Leaderboard commits in the repository
Leaderboard updated: RFD4-Proteina results added 2026-11-06, reformatted and consolidated into one entry on 2026-11-10/13. source
The most recent sourced event, not a claim that nothing changed since.
Next expectedThe maintainers have announced no next version or round.
Status● Active · Active: the leaderboard has new entries from 2025-02 through 2026-11.
ScaleThirty problems in three groups of ten by number of motif segments; 100 scaffolds required per problem.
  • Problems: 30 (paper v2)
  • Groups: 3 x 10 (paper v2)
  • Scaffolds per problem: 100 (paper v2)
  • Sequences redesigned per scaffold: 8 (paper v2)
  • Leaderboard entries: 15 (2026-11-16)
Wet-lab refreshStatic data · Purely computational evaluation; reference scaffolds come from experimental structures, but the evaluation itself has no wet lab.
Own wet lab needed to evaluate a new modelYes, to claim real generalization
Models reported3 counted (board 3)
Unverified fields† ranking.entries[0].date, ranking.entries[1].date, ranking.entries[2].date, ranking.entries[3].date
As of2026-11-16

What it measures

  • Task: Given a motif, generate fixed-length scaffolds; 100 per problem, judged by sequence redesign and refolding.
  • Truth: Computational oracle: geometric agreement between the refolded structure and the input motif.
  • Primary metric: MotifBench score (0-100)
  • Secondary metrics: Unique solutions after clustering; Novelty (TM-score against the PDB); Per-scaffold success rate
  • Version traps: State whether a number comes from the paper or the leaderboard, whether the indexed setting was used, and whether refolding used ESMFold or AlphaFold2.

Current ranking

Board

MetricMotifBench score (0-100)
Eval set30 motif-scaffolding problems, official ProteinMPNN + ESMFold pipeline
Snapshot
Boardhttps://github.com/blt2114/MotifBench
Retrieved2026-11-16

Caveat. Ranks follow the board table order (sorted by score, 15 entries); entries are submitter-run, not independently re-run. The top entry is not a site model.

Shown in the overview table:

RFD4-Proteina (indexed + medium thinking)
55.06 (± 1.35)
MotifBench score (0-100) · 30 motif-scaffolding problems, official ProteinMPNN + ESMFold pipeline
Board ·
Best site model: RFdiffusion3 (indexed) (#2)

RankModelScore (verbatim)Metric · eval set / protocolSource type · dateSource and note
1RFD4-Proteina (indexed + medium thinking)55.06 (± 1.35)MotifBench score (0-100)
30 motif-scaffolding problems, official ProteinMPNN + ESMFold pipeline
Board
†
source Not a site model; overall top-1.
2RFdiffusion3 (indexed)43.52 (± 1.35)MotifBench score (0-100)
30 motif-scaffolding problems, official ProteinMPNN + ESMFold pipeline
Board
†
source Submitter-run entry; best site model.
5La-Proteina29.75 (± 0.51)MotifBench score (0-100)
30 motif-scaffolding problems, official ProteinMPNN + ESMFold pipeline
Board
†
source Submitted by another generative-model team.
15ESM311.38MotifBench score (0-100)
30 motif-scaffolding problems, official ProteinMPNN + ESMFold pipeline
Board
†
source Last of the 15 entries.

Models reported

Grouped by relation type; each line shows its source type. Counted relations add up to 3. A pairing whose evidence rests only on this survey's own page is shown as unverified.

Official leaderboard or results (3)

  • ESM3 Board MotifBench submission leaderboard (2025-05 submission) · Entry on the MotifBench submission board (11.38, 2025-05). · evidence: link
  • La-Proteina Board · The board lists this model (29.75, 2026-07); the Chinese model page says there is no score, which is wrong. Resolution: The board lists La-Proteina (29.75, 2026-07), so the relation is official participation; the model page claiming no score is the error and must be fixed. · evidence: link
  • RFdiffusion2 / RFdiffusion3 Board official board entry (RFdiffusion3, 2026-08 snapshot) · The board lists RFdiffusion3 (43.52, 2026-08); the Chinese model page says no entry was found, which is wrong. Resolution: The board and this record agree in listing RFdiffusion3 (43.52, 2026-08) as official participation; the Chinese model page must be fixed. · evidence: link

Mentioned, not a pairing

  • · The Proteina-Complexa method paper does not mention this problem set; the earlier candidate list was wrong. no source link

Caveats

  1. Scores reflect in-silico self-consistency only, not expression, folding or function.
  2. Replicate standard deviations are about 0.39-0.47, and the authors treat gaps under about one point as unreliable.
  3. Paper and leaderboard give different RFdiffusion baselines (28.05 vs 21.95), so cite the protocol.
  4. Refolding with MSA-free AlphaFold2 instead of ESMFold drops the baseline from 28.1 to 22.5.
  5. Experimental reference scaffolds pass the evaluation in only 20 of 30 problems, which points to a limit of the evaluation itself.

How to use it

Take the 30 problems and the official pipeline from the repository, generate 100 fixed-length scaffolds per problem, score with the official scripts and submit to the leaderboard; about 36 GPU hours. Report whether the protocol is the paper's or the board's, whether the indexed setting was used, which refolder was chosen and how many replicates; gaps under about one point support no conclusion. Showing a design can be made still needs your own expression and assays.

Pairs well with: Adaptyv competitions / BenchBB · FoldBench · AIntibody

Sources

  1. MotifBench paper, arXiv 2502.12479 (preprint) ·
  2. Paper v2 full text (preprint)
  3. Official repository and leaderboard (leaderboard) ·

Corrections

2 corrections
FieldWasNowBasisFound
models_reportedRFdiffusion3 and La-Proteina: Chinese model pages state no MotifBench entry was foundboth are official-participation entries on the MotifBench board (43.52 and 29.75)source2026-11-16
models_reportedProteina-Complexa counted as a MotifBench pairingrecorded as not-a-pairing: the method paper never mentions the benchmarksource2026-11-16

Part of the AI for Bio benchmarks survey · all 44 benchmarks · open in the explorer. Record as of 2026-11-16.