AI for Bio · Benchmark 07

USPTO family (+PaRoutes)

Default data for retrosynthesis and reaction prediction; measures reproducing patent records, not lab feasibility.

bench-07Type BEvidence L2–L3Small molecules○ Frozen Not comparable中文

Overview

NameUSPTO family (+PaRoutes) · aliases: USPTO-50K, USPTO-MIT, USPTO-480K, MIT-480K, USPTO-FULL, USPTO-STEREO, PaRoutes
Full nameUSPTO patent reaction dataset family
Ambiguous names (not auto-linked)USPTO
FamilySeveral splits and a route set derived from one patent-text extraction; subsets differ in size, split and preprocessing, so numbers do not transfer across them.
  • Patent reaction extraction data 1976-2013 · †
  • USPTO-50K ·
  • Patent reactions 1976-Sep2016 (figshare) ·
  • USPTO-MIT 480K ·
  • PaRoutes · †
DomainSmall molecules
InclusionOriginal 30
TypeB · Static dataset family; PaRoutes adds route sets and scripts
Evidence levelL2–L3 · Random-split retrospective; OOD studies reach L3
PublisherNextMove Software, Novartis, Massachusetts Institute of Technology, AstraZeneca (Mixed)
First release · data
Basis: First public release of the patent reaction extraction data (NextMove blog, 2014-02); day not retrieved source
Other milestones
  • First public evidence: † · First public release of the patent reaction data source
  • Preprint v1: · USPTO-MIT 480K paper arXiv v1 source
  • Journal online: · USPTO-50K JCIM paper online source
Latest update · data
Basis: PaRoutes paper published in Digital Discovery (year only)
Newest member: PaRoutes route-benchmark paper in Digital Discovery (preprint reviews 2022-03-18); source data frozen at 2016-09. source
The most recent sourced event, not a claim that nothing changed since.
Next expectedSource data are frozen and no maintainer has announced a new version.
Status○ Frozen · Source data frozen at 2016; methods papers still report it
ScaleSource 3.3M reactions; USPTO-50K about 50k; MIT-480K 479,035; PaRoutes two sets of 10,000 routes.
  • Source reactions: 3,305,795 reactions (1976-Sep2016)
  • USPTO-50K reactions (10 classes): ~50,000 reactions (USPTO-50K)
  • MIT split (train/valid/test): 409,035 / 30,000 / 40,000 reactions (USPTO-MIT)
  • FULL raw reactions (preprocessing varies): 1,808,937 reactions (USPTO-FULL)
  • PaRoutes route sets: 2 × 10,000 routes (PaRoutes v1.0.0)
Wet-lab refreshStatic data · None; patent text mining, not experimental validation.
Own wet lab needed to evaluate a new modelYes, to claim real generalization
Models reported5 counted (self 4, modified protocol 1)
Unverified fields† family.members[0], family.members[4], ranking.entries[4], models_reported[2]
As of2026-11-16

What it measures

  • Task: Reaction outcome prediction, single-step retrosynthesis and multi-step route search on reactions mined from US patents.
  • Truth: Records extracted from patent text; no failed reactions, no yield guarantee, conditions often missing.
  • Primary metric: Top-k exact match (canonical SMILES)
  • Secondary metrics: Round-trip accuracy; Route solve rate (PaRoutes)
  • Version traps: Reaction-class disclosure, test-time augmentation, FULL pretraining and per-paper preprocessing all shift numbers; MIT has mixed and separated settings.

Current ranking

Not comparable

Metrictop-k exact match (%), canonical SMILES
Eval setUSPTO-50K, USPTO-MIT/480K, USPTO-FULL; settings differ (reaction class, augmentation, pretraining)
Snapshot
BoardNo official board
Retrieved2026-11-16

Caveat. No official board; Papers with Code closed. Numbers depend on class disclosure, augmentation and pretraining; ~5k test reactions, 1 point ~50 reactions; few seeds or CIs.

No headline entry: No official board and settings differ across papers, so entries are not comparable and no headline is given.

The entries below use different protocols, so they are listed, not ranked, and the overview table shows no score.

RankModelScore (verbatim)Metric · eval set / protocolSource type · dateSource and note
—RetroDFM-RWithheld: Not comparable: protocols and subsets differ, so no score is shown in the ranking row; per-protocol values stay on the entity page.top-k exact match (%), canonical SMILES
USPTO-50K, USPTO-MIT/480K, USPTO-FULL; settings differ (reaction class, augmentation, pretraining)
Self-reported
source Correction: the site cited arXiv 2510.16588 (that is C-SMILES); this model is arXiv 2507.17448; values came via a search summary, not a full read.
—Uni-Mol3Withheld: Not comparable: protocols and subsets differ, so no score is shown in the ranking row; per-protocol values stay on the entity page.top-k exact match (%), canonical SMILES
USPTO-50K, USPTO-MIT/480K, USPTO-FULL; settings differ (reaction class, augmentation, pretraining)
Self-reported
source Re-verified from the paper; the same tables show it below the Chemformer baseline on 50K retrosynthesis.
—ChemformerWithheld: Not comparable: protocols and subsets differ, so no score is shown in the ranking row; per-protocol values stay on the entity page.top-k exact match (%), canonical SMILES
USPTO-50K, USPTO-MIT/480K, USPTO-FULL; settings differ (reaction class, augmentation, pretraining)
Third-party
source Third-party tabulation (baseline row in the Uni-Mol3 paper), re-verified.
—C-SMILES (not a site model)Withheld: Not comparable: protocols and subsets differ, so no score is shown in the ranking row; per-protocol values stay on the entity page.top-k exact match (%), canonical SMILES
USPTO-50K, USPTO-MIT/480K, USPTO-FULL; settings differ (reaction class, augmentation, pretraining)
Self-reported
source Not a site model; highest 50K top-1 seen this round; the site's known/unknown-class split was not confirmed on the abstract.
—ether0Withheld: Incomplete provenance (no url) and a different protocol, so the honesty rule forbids showing a number.top-k exact match (%), canonical SMILES
USPTO-50K, USPTO-MIT/480K, USPTO-FULL; settings differ (reaction class, augmentation, pretraining)
Self-reported
No sourced date†
no source link Evaluated on its own reaction set, not an official USPTO test split; not comparable.

Models reported

Grouped by relation type; each line shows its source type. Counted relations add up to 5. A pairing whose evidence rests only on this survey's own page is shown as unverified.

Developer report, official protocol (4)

  • Chemformer Self-reported · USPTO-50K; the Uni-Mol3 re-quote is comparison only · evidence: link
  • RetroDFM-R Self-reported · USPTO-50K/FULL; with test-time augmentation noted · evidence: link
  • SciReasoner Self-reported · USPTO-50K single-step retrosynthesis, matrix note only, unverified †unverified · evidence: no source link
  • Uni-Mol3 Self-reported · USPTO-MIT/50K/Condition · evidence: link

Developer report, modified protocol (1)

  • ether0 Self-reported · Self-built reaction prediction, not official test set · evidence: link

Caveats

  1. Random splits leak templates (about 94% of USPTO-50K test reactions match training templates, per one report); exact match penalizes valid alternative routes; no failed reactions are recorded.
  2. Reaction-class disclosure, test-time augmentation and pretraining corpus can inflate numbers markedly; cross-paper comparison needs aligned settings.
  3. With about 5k test reactions, recent 1-3 point gains may be within noise (inference); most papers report no CIs.
  4. Template-free models drop sharply in OOD evaluations; single-step scores are reported to decouple from route-finding success.
  5. No evidence on lab feasibility of routes; only synthesis can establish it.

How to use it

Use the CC0 figshare source data or a method repo's preprocessed split; PaRoutes ships scripts. Report subset, reaction-class disclosure, augmentation, FULL pretraining, split (prefer new or time splits), seeds with CIs and a template-retrieval baseline; round-trip must name its forward model.

Pairs well with: GuacaMol (+MOSES) · PMO (Practical Molecular Optimization)

Sources

  1. Chemical reactions from US patents 1976-Sep2016 (figshare, CC0) (archive)
  2. USPTO-50K source paper (JCIM 2016) (journal)
  3. USPTO-MIT 480K paper (arXiv 1709.04555) (preprint)
  4. PaRoutes (Digital Discovery 2022) (journal)
  5. PaRoutes repository (repository)
  6. Retrosynthesis Prediction Revisited (NeurIPS 2022 workshop) (third-party paper)
  7. Retrosynthesis OOD evaluation (arXiv 2403.03960) (third-party paper)
  8. RetroDFM-R paper (arXiv 2507.17448) (preprint)

Corrections

3 corrections
FieldWasNowBasisFound
ranking.entries[model-retrodfm-r].urlarXiv 2510.16588 cited for RetroDFM-RarXiv 2507.17448 (2510.16588 is a different paper, C-SMILES)source2026-11-16
ranking.entries[C-SMILES].noteknown-class / unknown-class split as stated on the siteabstract gives a USPTO-50K figure and a USPTO-FULL figure; the known/unknown-class split is not confirmedsource2026-11-16
first_release.basisSource data dated 2012 (thesis)2014-02, first public data release; 2012 date unverifiedsource2026-11-16

Part of the AI for Bio benchmarks survey · all 44 benchmarks · open in the explorer. Record as of 2026-11-16.