AI for Bio · Benchmark 10

GuacaMol (+MOSES)

Entry-level molecular generation benchmark; computed oracles only, unmaintained, optimization side superseded by PMO.

bench-10Type DEvidence L2Molecule generation○ Frozen No ranking中文

Overview

NameGuacaMol (+MOSES) · aliases: GuacaMol, MOSES, Molecular Sets
Full nameGuacaMol molecular generation benchmark (with sibling MOSES)
Ambiguous names (not auto-linked)None
FamilyGuacaMol has distribution and goal-directed tasks; MOSES is a sibling benchmark from another team with distribution metrics only on ZINC data; scores do not transfer.
  • GuacaMol ·
  • MOSES ·
DomainMolecule generation
InclusionOriginal 30
TypeD · Computed-oracle benchmark, no official board
Evidence levelL2 · Retrospective on computed oracles
PublisherBenevolentAI, Insilico Medicine (Mixed)
First release · paper v1
Basis: GuacaMol paper arXiv v1 (MOSES arXiv v1 2018-11-29) source
Other milestones
  • Preprint v1: · GuacaMol arXiv 1811.09621 v1 source
  • Journal online: † · JCIM 2019 paper online source
  • Code release: · guacamol 0.3.1 uploaded to PyPI source
Latest update · maintenance only
Basis: guacamol 0.5.5 uploaded to PyPI
guacamol 0.5.5 uploaded to PyPI (maintenance); MOSES repo last commit 2023-06-22. source
The most recent sourced event, not a claim that nothing changed since.
Next expectedThe project is unmaintained and no maintainer has announced anything.
Status○ Frozen · Unmaintained; distribution tasks partly saturated
Scale5 distribution tasks plus 20 goal-directed tasks; ChEMBL 24 subset with 1,273,104 training SMILES.
  • Distribution-learning tasks: 5 tasks (GuacaMol)
  • Goal-directed tasks: 20 tasks (GuacaMol)
  • Train / valid / test SMILES: 1,273,104 / 79,568 / 238,706 SMILES (GuacaMol (ChEMBL 24))
  • MOSES molecules: 1,936,962 molecules (MOSES)
Wet-lab refreshStatic data · None; computational oracles and a ChEMBL training set.
Own wet lab needed to evaluate a new modelYes, to claim real generalization
Models reported0 counted
Unverified fields† release_milestones.journal_online, ranking.entries[0]
As of2026-11-16

What it measures

  • Task: Molecule generation: five distribution-learning tasks and 20 goal-directed tasks scored by computed oracles.
  • Truth: Public hand-written functions (similarity, descriptors, SMARTS, MPO); no experimental labels.
  • Primary metric: Goal-directed: summed weighted score over 20 tasks (max 20)
  • Secondary metrics: Distribution: Validity, Uniqueness, Novelty, KL, FCD; MOSES: SNN, Frag, Scaf, IntDiv
  • Version traps: No oracle-call budget, so totals grow with calls; GuacaMol and MOSES metrics differ; PMO oracles largely derive from GuacaMol and are not independent.

Current ranking

No ranking

Metricgoal-directed: sum of top-1/10/100 weighted scores over 20 tasks (max 20); distribution: Validity/Uniqueness/Novelty/KL/FCD
Eval setGuacaMol 20 goal-directed + 5 distribution-learning tasks (ChEMBL 24 subset); MOSES as sibling
Snapshot
BoardNo official board
Retrieved2026-11-16

Caveat. No official board and no site model reports a score; unmaintained, with optimization superseded by PMO's 10K-call protocol; distribution tasks are gameable.

No headline entry: No official board and no site model reports a GuacaMol score, so no headline is given.

RankModelScore (verbatim)Metric · eval set / protocolSource type · dateSource and note
1Graph GA20-task total 17.983 (original paper baselines: SMILES LSTM 17.340, SMILES GA 14.396, best-of-dataset 12.144, Graph MCTS 9.009)goal-directed: sum of top-1/10/100 weighted scores over 20 tasks (max 20); distribution: Validity/Uniqueness/Novelty/KL/FCD
GuacaMol 20 goal-directed + 5 distribution-learning tasks (ChEMBL 24 subset); MOSES as sibling
Self-reported
source second-hand Not a site model; from the original paper baselines, not re-verified this round; later higher totals are scattered self-reports without common protocol or seed variance.
—GenMolNo scoregoal-directed: sum of top-1/10/100 weighted scores over 20 tasks (max 20); distribution: Validity/Uniqueness/Novelty/KL/FCD
GuacaMol 20 goal-directed + 5 distribution-learning tasks (ChEMBL 24 subset); MOSES as sibling
Self-reported
source Reports no GuacaMol 20-task or MOSES score, using the data only for baselines; derived-oracle results sit under PMO.
—REINVENT 4No scoregoal-directed: sum of top-1/10/100 weighted scores over 20 tasks (max 20); distribution: Validity/Uniqueness/Novelty/KL/FCD
GuacaMol 20 goal-directed + 5 distribution-learning tasks (ChEMBL 24 subset); MOSES as sibling
Self-reported
source The paper cites GuacaMol and MOSES only as related work; the circulating number belongs to an earlier version re-tabulated by others; no pairing.

Models reported

Grouped by relation type; each line shows its source type. Counted relations add up to 0. A pairing whose evidence rests only on this survey's own page is shown as unverified.

None. No model among the 112 has a recorded pairing with this entry.

Caveats

  1. All truths are cheap computed oracles open to direct optimization; there is no oracle-call budget, and trivial edits can fill distribution tasks. PMO superseded it for optimization.
  2. In the original paper the top-scoring methods pass chemical quality filters less often, so score and chemistry quality pull apart.
  3. The 20 tasks are simply summed with no CIs or seed requirement; top gaps are likely within noise.
  4. Rediscovery and similarity tasks are largely solved; isomer and some MPO tasks still discriminate.
  5. No evidence links scores to real activity or synthesizability.

How to use it

Install guacamol, build the data and implement a generator interface; MOSES uses molsets. Prefer the PMO 10K-budget protocol with at least 5 seeds; report oracle calls, AUC Top-10, quality-filtered scores and any non-ChEMBL data.

Pairs well with: PMO (Practical Molecular Optimization) · USPTO family (+PaRoutes)

Sources

  1. GuacaMol paper (arXiv 1811.09621) (preprint)
  2. GuacaMol repository (repository)
  3. guacamol on PyPI (package registry)
  4. MOSES paper (arXiv 1811.12823) (preprint)
  5. MOSES repository (repository)
  6. PMO paper (arXiv 2206.12411) (third-party paper)
  7. Sample efficiency and filtering study (arXiv 2212.01385) (third-party paper)

Corrections

1 correction
FieldWasNowBasisFound
latest_versionno formal releasePyPI releases guacamol 0.3.1 (2018-11-28) to 0.5.5 (2023-09-19)source2026-11-16

Part of the AI for Bio benchmarks survey · all 44 benchmarks · open in the explorer. Record as of 2026-11-16.