Overview
| Name | GuacaMol (+MOSES) · aliases: GuacaMol, MOSES, Molecular Sets |
|---|---|
| Full name | GuacaMol molecular generation benchmark (with sibling MOSES) |
| Ambiguous names (not auto-linked) | None |
| Family | GuacaMol has distribution and goal-directed tasks; MOSES is a sibling benchmark from another team with distribution metrics only on ZINC data; scores do not transfer.
|
| Domain | Molecule generation |
| Inclusion | Original 30 |
| Type | D · Computed-oracle benchmark, no official board |
| Evidence level | L2 · Retrospective on computed oracles |
| Publisher | BenevolentAI, Insilico Medicine (Mixed) |
| First release | · paper v1 Basis: GuacaMol paper arXiv v1 (MOSES arXiv v1 2018-11-29) source |
| Other milestones | |
| Latest update | · maintenance only Basis: guacamol 0.5.5 uploaded to PyPI guacamol 0.5.5 uploaded to PyPI (maintenance); MOSES repo last commit 2023-06-22. source The most recent sourced event, not a claim that nothing changed since. |
| Next expected | The project is unmaintained and no maintainer has announced anything. |
| Status | ○ Frozen · Unmaintained; distribution tasks partly saturated |
| Scale | 5 distribution tasks plus 20 goal-directed tasks; ChEMBL 24 subset with 1,273,104 training SMILES.
|
| Wet-lab refresh | Static data · None; computational oracles and a ChEMBL training set. |
| Own wet lab needed to evaluate a new model | Yes, to claim real generalization |
| Models reported | 0 counted |
| Unverified fields | † release_milestones.journal_online, ranking.entries[0] |
| As of | 2026-11-16 |
What it measures
- Task: Molecule generation: five distribution-learning tasks and 20 goal-directed tasks scored by computed oracles.
- Truth: Public hand-written functions (similarity, descriptors, SMARTS, MPO); no experimental labels.
- Primary metric: Goal-directed: summed weighted score over 20 tasks (max 20)
- Secondary metrics: Distribution: Validity, Uniqueness, Novelty, KL, FCD; MOSES: SNN, Frag, Scaf, IntDiv
- Version traps: No oracle-call budget, so totals grow with calls; GuacaMol and MOSES metrics differ; PMO oracles largely derive from GuacaMol and are not independent.
Current ranking
No ranking
| Metric | goal-directed: sum of top-1/10/100 weighted scores over 20 tasks (max 20); distribution: Validity/Uniqueness/Novelty/KL/FCD |
|---|---|
| Eval set | GuacaMol 20 goal-directed + 5 distribution-learning tasks (ChEMBL 24 subset); MOSES as sibling |
| Snapshot | |
| Board | No official board |
| Retrieved | 2026-11-16 |
Caveat. No official board and no site model reports a score; unmaintained, with optimization superseded by PMO's 10K-call protocol; distribution tasks are gameable.
No headline entry: No official board and no site model reports a GuacaMol score, so no headline is given.
| Rank | Model | Score (verbatim) | Metric · eval set / protocol | Source type · date | Source and note |
|---|---|---|---|---|---|
| 1 | Graph GA | 20-task total 17.983 (original paper baselines: SMILES LSTM 17.340, SMILES GA 14.396, best-of-dataset 12.144, Graph MCTS 9.009) | goal-directed: sum of top-1/10/100 weighted scores over 20 tasks (max 20); distribution: Validity/Uniqueness/Novelty/KL/FCD GuacaMol 20 goal-directed + 5 distribution-learning tasks (ChEMBL 24 subset); MOSES as sibling | Self-reported | source second-hand Not a site model; from the original paper baselines, not re-verified this round; later higher totals are scattered self-reports without common protocol or seed variance. |
| — | GenMol | No score | goal-directed: sum of top-1/10/100 weighted scores over 20 tasks (max 20); distribution: Validity/Uniqueness/Novelty/KL/FCD GuacaMol 20 goal-directed + 5 distribution-learning tasks (ChEMBL 24 subset); MOSES as sibling | Self-reported | source Reports no GuacaMol 20-task or MOSES score, using the data only for baselines; derived-oracle results sit under PMO. |
| — | REINVENT 4 | No score | goal-directed: sum of top-1/10/100 weighted scores over 20 tasks (max 20); distribution: Validity/Uniqueness/Novelty/KL/FCD GuacaMol 20 goal-directed + 5 distribution-learning tasks (ChEMBL 24 subset); MOSES as sibling | Self-reported | source The paper cites GuacaMol and MOSES only as related work; the circulating number belongs to an earlier version re-tabulated by others; no pairing. |
Models reported
Grouped by relation type; each line shows its source type. Counted relations add up to 0. A pairing whose evidence rests only on this survey's own page is shown as unverified.
None. No model among the 112 has a recorded pairing with this entry.
Caveats
- All truths are cheap computed oracles open to direct optimization; there is no oracle-call budget, and trivial edits can fill distribution tasks. PMO superseded it for optimization.
- In the original paper the top-scoring methods pass chemical quality filters less often, so score and chemistry quality pull apart.
- The 20 tasks are simply summed with no CIs or seed requirement; top gaps are likely within noise.
- Rediscovery and similarity tasks are largely solved; isomer and some MPO tasks still discriminate.
- No evidence links scores to real activity or synthesizability.
How to use it
Install guacamol, build the data and implement a generator interface; MOSES uses molsets. Prefer the PMO 10K-budget protocol with at least 5 seeds; report oracle calls, AUC Top-10, quality-filtered scores and any non-ChEMBL data.
Pairs well with: PMO (Practical Molecular Optimization) · USPTO family (+PaRoutes)
Sources
- GuacaMol paper (arXiv 1811.09621) (preprint)
- GuacaMol repository (repository)
- guacamol on PyPI (package registry)
- MOSES paper (arXiv 1811.12823) (preprint)
- MOSES repository (repository)
- PMO paper (arXiv 2206.12411) (third-party paper)
- Sample efficiency and filtering study (arXiv 2212.01385) (third-party paper)
Corrections
1 correction
| Field | Was | Now | Basis | Found |
|---|---|---|---|---|
| latest_version | no formal release | PyPI releases guacamol 0.3.1 (2018-11-28) to 0.5.5 (2023-09-19) | source | 2026-11-16 |
Part of the AI for Bio benchmarks survey · all 44 benchmarks · open in the explorer. Record as of 2026-11-16.