AI for Bio · Benchmark 26

FLIP / FLIP2

Supervised protein fitness benchmark using deliberately shifted splits to test low-resource learning and extrapolation.

bench-26Type BEvidence L2–L3Proteins◐ Low-frequency Not comparable中文

Overview

NameFLIP / FLIP2 · aliases: FLIP, FLIP2, FLIPv2
Full nameFitness Landscape Inference for Proteins (FLIP) and its second generation
Ambiguous names (not auto-linked)flip
FamilyThe two generations share a repository and splitting philosophy but differ in datasets and splits, so numbers do not compare across generations.
  • FLIP ·
  • FLIP2 · †
DomainProteins
InclusionOriginal 30
TypeB · Static dataset suite for supervised extrapolation, with no leaderboard
Evidence levelL2–L3 · Deliberate extrapolation splits, but within one protein and one assay
PublisherFLIP benchmark maintainers, an academic consortium (Academic lab)
First release · paper v1
Basis: The first generation's bioRxiv v1, published the same year in the NeurIPS Datasets and Benchmarks track. source
Other milestones
  • Preprint v1: · First generation, bioRxiv v1 source
  • Code release: † · Second-generation release pull request merged into the main branch source
Latest update† partly verified · new version
Basis: Release merge record in the official repository
The second-generation release was merged into the main branch, bringing data and code; later commits only refresh documentation pages.
Note: The merge date was not re-retrieved this round, and the archive date is a separate event source
The most recent sourced event, not a claim that nothing changed since.
Next expectedThe maintainers have not announced a third generation or new splits.
Status◐ Low-frequency · The first generation is maintained infrequently; the second shipped in 2026; no official board
ScaleFirst generation: 3 landscapes and 15 splits; second generation: 7 new datasets and 16 predefined splits.
  • Landscapes: 3 (FLIP)
  • Predefined splits: 15 (FLIP)
  • Variants across the three landscapes: 8733 / 82583 / about 17,000 (FLIP)
  • New datasets: 7 (FLIP2)
  • Predefined splits: 16 (FLIP2)
Wet-lab refreshStatic data · No new experiments; reuses published deep mutational scans and thermostability data.
Own wet lab needed to evaluate a new modelNo, run it as a download
Models reported2 counted (self 2)
Unverified fields† latest_update, release_milestones.code_release, family.members[1], scale.items[2]
As of2026-11-16

What it measures

  • Task: Supervised prediction of protein fitness under deliberately shifted splits, testing low-resource learning and extrapolation.
  • Truth: Published deep mutational scan readouts and thermostability measurements, organised per protein and assay.
  • Primary metric: Spearman rho per split
  • Secondary metrics: Variance across splits and sensitivity to the training recipe
  • Version traps: State the generation, split name and training recipe: some older splits are flagged by the authors as unsuitable for comparison, and averaging splits hides differences.

Current ranking

Not comparable

MetricSpearman rho, per split
Eval setFLIP splits (GB1, AAV, Meltome); FLIP2 adds 7 datasets
Snapshot
BoardNo official board
Retrieved2026-11-16

Caveat. There is no official online board and all numbers come from per-task tables in papers; a third-party re-evaluation shows recipe noise exceeding model differences, and random splits do not mix with extrapolation splits.

No headline entry: There is no official board and the entries differ in split and recipe, so no headline number is given.

The entries below use different protocols, so they are listed, not ranked, and the overview table shows no score.

RankModelScore (verbatim)Metric · eval set / protocolSource type · dateSource and note
—xTrimoPGLM (LoRA fine-tuned, not zero-shot)Withheld: Not comparable: splits and training recipes differ, so the ranking row shows no score.Spearman rho, per split
FLIP splits (GB1, AAV, Meltome); FLIP2 adds 7 datasets
Self-reported
source Fine-tuned on the random control split, near the ceiling, not an extrapolation split
—SaProtWithheld: Not comparable: splits and training recipes differ, so the ranking row shows no score.Spearman rho, per split
FLIP splits (GB1, AAV, Meltome); FLIP2 adds 7 datasets
Self-reported
source Self-report on a thermostability split; another dataset is figure-only with no quotable number
—SaProtWithheld: Not comparable: splits and training recipes differ, so the ranking row shows no score.Spearman rho, per split
FLIP splits (GB1, AAV, Meltome); FLIP2 adds 7 datasets
Third-party
source Third-party re-evaluation on extrapolation splits, a different protocol from the self-reports

Models reported

Grouped by relation type; each line shows its source type. Counted relations add up to 2. A pairing whose evidence rests only on this survey's own page is shown as unverified.

Developer report, official protocol (2)

  • SaProt Self-reported Meltome human-cell (called Thermostability in the paper); AAV 2-vs-rest · Self-reported on two datasets; a third-party re-evaluation also exists · evidence: link
  • xTrimoPGLM Self-reported · Self-reported after fine-tuning on the random control split · evidence: link

Caveats

  1. Scores swing with training recipe and split, and averaging splits hides large differences.
  2. Extrapolation stays within one protein and one assay, not across families.
  3. The second generation reports that simple models often match fine-tuned protein language models.
  4. There is no official online board, reuse protocols vary, and larger suites have overtaken its influence.
  5. Some first-generation splits are flagged by the authors as unsuitable for comparison or outdated.

How to use it

Download one generation's data and predefined splits, train per split and report Spearman rho; it runs statically with no wet lab of your own. State the generation, split name, training recipe and seed variance, report per split rather than averaged, and compare against simple baselines. Cross-protein generalization needs a fresh assay. Pair with ProteinGym and the few-shot setting of FS-Mol.

Pairs well with: ProteinGym · CAGI · FS-Mol · RNAGym

Sources

  1. First-generation preprint (preprint)
  2. Official repository and data documentation (repository)
  3. Second-generation data archive (archive)
  4. Second-generation preprint (preprint)
  5. Third-party re-evaluation on these splits (third-party paper)

Corrections

2 corrections
FieldWasNowBasisFound
FLIP2 release date2026-05archive publication 2026-01-30; repository release commit 2026-03-03; release pull request merged 2026-04-23source2026-11-16
ranking.entriesper-model scores shown in the ranking rowscores withheld; splits and training recipes differsource2026-11-16

Part of the AI for Bio benchmarks survey · all 44 benchmarks · open in the explorer. Record as of 2026-11-16.