Overview
| Name | FLIP / FLIP2 · aliases: FLIP, FLIP2, FLIPv2 |
|---|---|
| Full name | Fitness Landscape Inference for Proteins (FLIP) and its second generation |
| Ambiguous names (not auto-linked) | flip |
| Family | The two generations share a repository and splitting philosophy but differ in datasets and splits, so numbers do not compare across generations.
|
| Domain | Proteins |
| Inclusion | Original 30 |
| Type | B · Static dataset suite for supervised extrapolation, with no leaderboard |
| Evidence level | L2–L3 · Deliberate extrapolation splits, but within one protein and one assay |
| Publisher | FLIP benchmark maintainers, an academic consortium (Academic lab) |
| First release | · paper v1 Basis: The first generation's bioRxiv v1, published the same year in the NeurIPS Datasets and Benchmarks track. source |
| Other milestones | |
| Latest update | † partly verified · new version Basis: Release merge record in the official repository The second-generation release was merged into the main branch, bringing data and code; later commits only refresh documentation pages. Note: The merge date was not re-retrieved this round, and the archive date is a separate event source The most recent sourced event, not a claim that nothing changed since. |
| Next expected | The maintainers have not announced a third generation or new splits. |
| Status | ◐ Low-frequency · The first generation is maintained infrequently; the second shipped in 2026; no official board |
| Scale | First generation: 3 landscapes and 15 splits; second generation: 7 new datasets and 16 predefined splits.
|
| Wet-lab refresh | Static data · No new experiments; reuses published deep mutational scans and thermostability data. |
| Own wet lab needed to evaluate a new model | No, run it as a download |
| Models reported | 2 counted (self 2) |
| Unverified fields | † latest_update, release_milestones.code_release, family.members[1], scale.items[2] |
| As of | 2026-11-16 |
What it measures
- Task: Supervised prediction of protein fitness under deliberately shifted splits, testing low-resource learning and extrapolation.
- Truth: Published deep mutational scan readouts and thermostability measurements, organised per protein and assay.
- Primary metric: Spearman rho per split
- Secondary metrics: Variance across splits and sensitivity to the training recipe
- Version traps: State the generation, split name and training recipe: some older splits are flagged by the authors as unsuitable for comparison, and averaging splits hides differences.
Current ranking
Not comparable
| Metric | Spearman rho, per split |
|---|---|
| Eval set | FLIP splits (GB1, AAV, Meltome); FLIP2 adds 7 datasets |
| Snapshot | |
| Board | No official board |
| Retrieved | 2026-11-16 |
Caveat. There is no official online board and all numbers come from per-task tables in papers; a third-party re-evaluation shows recipe noise exceeding model differences, and random splits do not mix with extrapolation splits.
No headline entry: There is no official board and the entries differ in split and recipe, so no headline number is given.
The entries below use different protocols, so they are listed, not ranked, and the overview table shows no score.
| Rank | Model | Score (verbatim) | Metric · eval set / protocol | Source type · date | Source and note |
|---|---|---|---|---|---|
| — | xTrimoPGLM (LoRA fine-tuned, not zero-shot) | Withheld: Not comparable: splits and training recipes differ, so the ranking row shows no score. | Spearman rho, per split FLIP splits (GB1, AAV, Meltome); FLIP2 adds 7 datasets | Self-reported | source Fine-tuned on the random control split, near the ceiling, not an extrapolation split |
| — | SaProt | Withheld: Not comparable: splits and training recipes differ, so the ranking row shows no score. | Spearman rho, per split FLIP splits (GB1, AAV, Meltome); FLIP2 adds 7 datasets | Self-reported | source Self-report on a thermostability split; another dataset is figure-only with no quotable number |
| — | SaProt | Withheld: Not comparable: splits and training recipes differ, so the ranking row shows no score. | Spearman rho, per split FLIP splits (GB1, AAV, Meltome); FLIP2 adds 7 datasets | Third-party | source Third-party re-evaluation on extrapolation splits, a different protocol from the self-reports |
Models reported
Grouped by relation type; each line shows its source type. Counted relations add up to 2. A pairing whose evidence rests only on this survey's own page is shown as unverified.
Developer report, official protocol (2)
- SaProt Self-reported Meltome human-cell (called Thermostability in the paper); AAV 2-vs-rest · Self-reported on two datasets; a third-party re-evaluation also exists · evidence: link
- xTrimoPGLM Self-reported · Self-reported after fine-tuning on the random control split · evidence: link
Caveats
- Scores swing with training recipe and split, and averaging splits hides large differences.
- Extrapolation stays within one protein and one assay, not across families.
- The second generation reports that simple models often match fine-tuned protein language models.
- There is no official online board, reuse protocols vary, and larger suites have overtaken its influence.
- Some first-generation splits are flagged by the authors as unsuitable for comparison or outdated.
How to use it
Download one generation's data and predefined splits, train per split and report Spearman rho; it runs statically with no wet lab of your own. State the generation, split name, training recipe and seed variance, report per split rather than averaged, and compare against simple baselines. Cross-protein generalization needs a fresh assay. Pair with ProteinGym and the few-shot setting of FS-Mol.
Pairs well with: ProteinGym · CAGI · FS-Mol · RNAGym
Sources
- First-generation preprint (preprint)
- Official repository and data documentation (repository)
- Second-generation data archive (archive)
- Second-generation preprint (preprint)
- Third-party re-evaluation on these splits (third-party paper)
Corrections
2 corrections
Part of the AI for Bio benchmarks survey · all 44 benchmarks · open in the explorer. Record as of 2026-11-16.