This is a study of the AI and agent benchmark supply chain: leaderboards built by data-service vendors versus public academic standards, the US compared with China, and a methodological trap most people have not noticed, namely the harness (or scaffold) used during evaluation.
"Model M scores 65 on leaderboard X" carries no information on its own, because you do not know what would happen with a different harness. Comparing two such numbers compares two systems, not two models.
1. One benchmark, five independent dimensions
Before looking at any data, we set up an analytical framework. Every judgement below follows these five dimensions. They are independent of each other, and the combination of values a benchmark takes across them largely determines its business logic, its ceiling on credibility, and whether it can be used for objective comparison.
| Dimension | Values | What it determines |
|---|---|---|
| ① Who sets the questions | Commercial data/environment company · academia or research institute · big-lab in-house (a lab sets questions to test itself and its peers) · platform/marketplace (sets no questions, only provides infrastructure) | Whether a commercial motive exists, and whether there is a "referee who is also a player" conflict of interest |
| ② What is evaluated | Static knowledge/reasoning · agentic ability (multi-step, tool use) · vertical professional skill (investment banking, law, medicine, research) | Whether the questions can be multiple choice, which indirectly sets cost and how far grading can be automated |
| ③ How it is evaluated | Pure API calls · requires a dedicated harness/environment · requires human review | Whether the "change the scaffold and the score changes" trap applies (see §5) |
| ④ Openness | Fully open · partly public (to resist training contamination) · fully private | Whether third parties can reproduce results, and whether the question bank will be gamed |
| ⑤ Regional ecosystem | US · China, each with its own mechanism for what counts as "accepted" | Cited is not the same as trusted; the two regions build trust in entirely different ways (see §3 and §9) |
Of the five, ① determines motive, ③ determines methodological risk and ⑤ determines the credibility ceiling. Those three are the threads this piece keeps returning to; ② and ④ are mostly descriptive labels.
2. Supply-chain economics: how a task is produced, priced and sold
To understand why so many commercial companies are rushing to build benchmarks, you first need the economics of the business itself. A benchmark is only the visible tip of the supply chain; underneath sits the whole industry of training data and RL environments.
The four components of a sellable environment
- An interactive world: a Docker container or micro-VM running real software or a replica of it.
- Tasks: an initial state plus a prompt. These are not one-off runs but designed, reusable questions.
- A grader (verifier): the most expensive and most central part. Three tiers, from most to least expensive: programmatic verification (unit tests, state checks) > rubric scoring > LLM judge.
- Resettable and parallel: training needs dozens of rollouts per task, with thousands of containers running concurrently.
The cost goes into building the initial state and the grader. Experts must find real, difficult scenarios where frontier models genuinely fail, construct a clean reproducible environment, and write verification tests the model cannot cheat. This is also why benchmarks that need a harness (about 47% by the count in §4) are generally much more expensive and harder to build than pure knowledge tests.
Pricing structure
| Unit | Price range | Notes |
|---|---|---|
| Single task | USD 200–2,000 | The mainstream range; complex software-engineering tasks occasionally reach USD 20,000 |
| Website/UI gym replica | About USD 20,000 each | OpenAI reportedly buys browser environments at this price |
| High-fidelity product replica | Up to about USD 300,000 each | Enterprise software replicas of the kind Fleet builds |
| Quarterly contract | Six to seven figures USD per quarter | The common form between labs and top vendors |
| Exclusivity premium | 4–5× the non-exclusive price | Labs do not want competitors to get the same data |
Four production models
| Model | Examples | Characteristics |
|---|---|---|
| Handcrafted boutique | Mechanize | Full-time experts polish each task; depth first, "few but deep" |
| Assembly-line replicas | Fleet, Scale RL Environments | Software replica templates plus tooling; one base spawns thousands of tasks and can be sold to several buyers |
| Platform + marketplace | HUD, Prime Intellect | Build no environments themselves; provide SDKs, hosting and matchmaking, "selling shovels to gold miners" |
| Expert-network question writing | Mercor, Handshake, Surge | Talent pools find experts to write questions, graded by humans or rubrics; moving toward automated environments |
Domain distribution: coding is the holy grail, and science went from "empty" to "crowded but without a standard"
Coding is called the holy grail: long-horizon reasoning, tool use, statefulness, error recovery and dense feedback (compiling and testing act as the reward). It is where investment is most concentrated, followed by computer use and enterprise software. Medicine, law and finance rely mainly on expert-written questions.
An earlier version of this section judged that "science is almost empty." That conclusion is out of date and needs correcting. In the first half of 2026 at least 7–9 new research-agent benchmarks appeared in quick succession (AutoResearchBench, ResearchClawBench, SciAgentArena from Microsoft, MLR-Bench, AIRS-Bench, HeurekaBench and others; see §6), from institutions as different as Meta, Microsoft Research, EPFL and the Shanghai AI Laboratory (InternScience). Questions are no longer scarce.
But the bottleneck has moved from "nobody writes the questions" to "nobody agrees." These benchmarks each cover a different slice of the research lifecycle (some test ideation and planning, some test execution, and they do not connect), and none has been adopted by several labs together or written into a system card. The root cause is no longer a lack of automated verifiers (rubrics plus LLM judges can already get around that). It is that research tasks are so open-ended and expert rubrics so subjective that the same question can swing widely in score depending on who grades it. This echoes the harness sensitivity in §5, and it is worse for research benchmarks (see the GAIA case in §6). The gap is still being filled not by annotation companies but by "the lab as reward function" (Periodic Labs) and "research software connected directly to the lab" (Anthropic × Benchling), which bypass the benchmark layer and use real experimental results as the training signal.
3. Who builds benchmarks, and why
AI benchmarks now have two entirely different kinds of producer. One is data and environment vendors. They sell training data and RL environments, and also set and grade their own tests; evaluation is essentially the first step of the sales funnel: diagnose where a model is weak, then sell you the data or environment to fix it. The other is academic and research institutions, whose goal is a shared public yardstick for the whole industry, with no commercial conversion attached. Being cited is their only "return".
Each kind has evolved its own system in the US and in China, giving four quadrants:
US · Vendor-built
- Scale AI: SEAL Leaderboards (including SWE-bench Pro)
- Surge AI: Hemingway-bench / AdvancedIF / GDP.pdf / Riemann-bench
- Mercor: the APEX series (-1 / -SWE / -Agents / -Accounting)
- Handshake AI: BankerToolBench / ATLAS-Finance / VIALS
- HUD: OSWorld-Verified / SheetBench-50
- Fleet and Mechanize have no separately branded public leaderboard
US · Academic / community standards
- MMLU / MMLU-Pro, GPQA Diamond, HLE
- GSM8K, MATH, AIME
- HumanEval, MBPP, SWE-bench Verified, LiveCodeBench
- AgentBench, WebArena, τ-bench
- Chatbot Arena/LMArena, AlpacaEval, MT-Bench
China · Vendor-built
- UniPat AI: USD 300M Series A led by Alibaba in Sep 2026, valued at USD 2.5B
- Humanlaya (formerly Rang Zhineng): One-Million Bench / BiomniBench
- HongShan (formerly Sequoia China) xbench: ScienceQA / DeepSearch / AgentIF / BabyVision
- Zhiyan Huisheng: Tencent has invested; little public information
- Haitian Ruisheng (SpeechOcean) / Datatang / Testin Data: have evaluation services but no productised public leaderboard
China · Academic / national-team standards
- C-Eval (Shanghai Jiao Tong + Tsinghua + Edinburgh)
- CLUE / CLUEWSC, CMMLU
- BAAI's FlagEval
- Shanghai AI Laboratory's OpenCompass (Sinan)
- SuperCLUE, Feixian Zhineng's ReLE
Laid on top of each other, the four quadrants show that the US and China are structurally the same: what is truly used across vendors is always the non-commercial academic standard, and vendor leaderboards at most fill niches and serve as sales endorsements. The difference is maturity. US vendor leaderboards have made it into some top labs' official system cards (even if only as customers showing off results), while the Chinese companies that started in 2025 still earn trust mainly from capital markets and have not yet been formally adopted in a leading model company's technical report.
Why vendors build their own benchmarks: an income statement
Not every vendor has one. Of the old four (Scale, Surge, Mercor, Handshake) plus the newer three, five (the old four and HUD) have built and branded public benchmarks. Fleet and Mechanize have not. They are precisely the two with the smallest teams and the most boutique, vertical strategy, putting all their limited expert capacity into the core product.
Benefits
- Free authoritative endorsement: a citation in a lab's system card works like third-party certification.
- Proof of production capability: being able to write questions that frontier models fail is itself a demonstration of expertise to buyers.
- Defining the standard means owning the conversation: labs that want a score cannot avoid you.
- Feeds the data flywheel: failure modes seen on the leaderboard directly shape the next batch of training tasks.
- Differentiation: APEX focuses on professional tasks and BankerToolBench on investment banking, avoiding sameness.
Costs
- The referee-player paradox: selling data while grading; recall that Scale's customers pulled orders en masse after Meta's investment raised doubts about its neutrality.
- High maintenance: question banks get gamed and overfit, so questions must be rotated continually to prevent leakage.
- Reputation is fragile: if questions are revealed to have leaked to one lab's training, the damage far exceeds the benefit.
- Distraction from the core business: it is essentially marketing, with a risk of misallocated resources.
- Backfire: it is awkward when your own customer's model ranks badly on your leaderboard.
A self-built benchmark is more of a brand-premium play for vendors that are already large and deep in experts than a standard feature of data and environment suppliers. That Fleet and Mechanize skip it confirms that a leaderboard is a nice-to-have, not a survival requirement.
4. The full catalog: 64 benchmarks, one by one
Each benchmark is recorded on the same six fields: size of question bank, openness, capability evaluated, host, citation by model companies, and evaluation method (pure API or dedicated harness). They are grouped by the four quadrants; click to expand.
US · Vendor-built16
| Benchmark | Size | Openness | Capability | Host | Cited by | Method |
|---|---|---|---|---|---|---|
| Scale SEAL Leaderboards | ~1,000 questions per sub-board | Private | Code / Spanish / instruction following / math | Scale AI | Anthropic Opus 4.6 | Human review |
| Scale SWE-bench Pro | 1,865 (731 public + 858 held-out + 276 private) | Partly public | Engineering fixes in real codebases | Scale AI | None found | Needs harness |
| Surge Hemingway-bench | Not disclosed | Private | Creative / business writing quality | Surge AI | None found | Human review |
| Surge AdvancedIF | 1,600+ prompts | Private | Complex multi-turn instruction following | Surge AI (with Meta MSL) | No formal MSL citation found | Expert rubric |
| Surge GDP.pdf | 100 (10 professional fields) | Private | Understanding real professional documents | Surge AI | OpenAI GPT-5.6; Anthropic Fable5/Mythos5 | Human review |
| Surge Riemann-bench | 25 | Methodology public | Research-level frontier mathematics | Surge AI | Anthropic Fable5/Mythos5 | API + automatic checking |
| Mercor APEX-1 | 400 hidden + 100 open on HF | Partly public | Investment banking / consulting / law firms / doctors | Mercor | None found | Pure API + judge |
| Mercor APEX-SWE | 200 (50 open on HF) | Partly public | Real software integration / debugging | Mercor | None found | Needs harness |
| Mercor APEX-Agents | 480 | Fully open | Long-horizon cross-app professional-services agent | Mercor (Archipelago) | Anthropic Opus 4.6 | Needs harness |
| Mercor APEX-Accounting | 160 tasks (10 "worlds", each with an accounting system + spreadsheets + PDFs) | Tasks CC-BY; Archipelago environment open source | Agentic accounting tasks (built with Ramp) | Mercor | None found | Needs harness (Archipelago) |
| Handshake BankerToolBench | 100 | Fully open | End-to-end investment-banking workflows | Handshake AI Research | None found | Harbor |
| Handshake ATLAS-Finance | 100 (13 financial simulation environments) | Fully open | Multi-app collaboration at financial institutions | Handshake AI Research | None found | Harbor |
| Handshake VIALS | 161 | Fully open | Life-science visual understanding (VQA), part of the ATLAS suite family | Handshake AI Research | None found; released Aug 2026 (arXiv 2608.21357) | API + LLM judge; best model still <30% accurate |
| HUD OSWorld-Verified | 369+ desktop tasks | Public | Operating real desktop environments | Created by XLang Labs, maintained by HUD | Widely tested by OpenAI, Google, Claude and others | hud-python SDK |
| HUD SheetBench-50 | 50 | Fully open | Spreadsheet-analyst skills; CFO-level experts from PwC, Cisco, Schwab and Fannie Mae helped review | HUD × Sepal AI (YC S24) | None found; used internally by PwC and others | hud-python SDK |
| Mechanize GBA Eval | One task (24-hour long horizon) | Methodology public | Extreme long-horizon software engineering | Mechanize | Self-tested; no reverse citation found | Docker + goose |
An interesting M&A crossover: Sepal AI (YC S24, a network of 20,000+ experts), which helped HUD build SheetBench-50, was acquired by Mercor on Feb 6, 2026. In other words, the team that built an evaluation set for one vendor was later bought by that vendor's competitor.
China · Vendor-built13
| Benchmark | Size | Openness | Capability | Host | Cited by | Method |
|---|---|---|---|---|---|---|
| xbench-ScienceQA | 100 | Fully public | Science Q&A / reasoning | xbench.org (HongShan) | None found | Pure API |
| xbench-DeepSearch | 100 | Fully public | Deep search of the Chinese-language internet | xbench.org | None found | Fixed ReAct agent (5 tools) |
| AgentIF-OneDay | 104 tasks / 767 scoring points | Fully public | Agent instruction following in everyday scenarios | xbench.org | None found | Any architecture; graded on deliverables |
| BabyVision | 388 + 20-question Mini | Code public but no licence file | Pure visual understanding / reasoning (arXiv 2601.06521) | xbench.org × UniPat | Joint PR case | Pure multimodal API |
| UniPat Monthly-SWEBench | ~100 per month / ~20 repos, multilingual | No public repo (leaderboard page only) | Real GitHub issues, refreshed monthly against contamination | unipat.ai | None found | Harbor + Terminus-2 |
| UniPat SaaS-Bench | 23 systems / 106 tasks / 3,971 checkpoints | Open source, Apache-2.0 | GUI agents operating real SaaS (arXiv 2605.15777) | unipat.ai | None found | Own Docker + browser-use |
| UniPat DeepTerminalBench | 50 tasks | Only mentioned in the Terminal-X repo description; no separate repo | Single-turn deep engineering (the "depth" branch of Terminal-X) | unipat.ai | None found | Terminal-Bench 2.0 format + Harbor |
| UniPat RoadmapBench | 115 tasks / 17 codebases / 5 languages | Open source, MIT | Coding agents upgrading versions to spec (long horizon) | unipat.ai | None found | Harbor + OpenHands |
| UniPat EvoCode-Bench | 26 tasks / 5–15 rounds / 227 evaluation rounds | Open source, MIT | Multi-round coding under changing requirements (arXiv 2605.24110) | unipat.ai | None found | Harbor multi-turn fork |
| UniPat ClawBench | Not disclosed | Open source; ownership unclear | Browser agents on real websites | NAIL Group (UniPat a participant) | None found | OpenClaw by default |
| Humanlaya One-Million Bench | 400 (200 Chinese + 200 English) | Open source | Expert tasks across 5 domains and 37 subdomains | humanlaya.com | Self-tested on 50+ models | API + negative rubric |
| Humanlaya BiomniBench | 100 tasks (50 public + 50 private) | Partly open | Process-level evaluation of biomedical research agents | xbench.org, with Phylo/Humanlaya | None found; companion to Stanford's Biomni | Harbor + Terminus-2 |
| Humanlaya ExcelBench | After further checking: Humanlaya's website, GitHub and Hugging Face show no separately released ExcelBench question bank or paper. It appears to be a media label for "Excel modelling tasks" rather than a formal product, so it is no longer counted as a separate benchmark. | |||||
| Haitian Ruisheng internal evaluation system | No public size | Not open, B2B | General language + safety/reliability (6 capabilities, 19 sub-capabilities) | Haitian Ruisheng (DOTS-LLM) | None found | API + human review (inferred) |
There are two different SaaS(-)Bench papers. ① SaaS-Bench (arXiv 2605.15777) is UniPat's, Can Computer-Use Agents Leverage Real-World SaaS to Solve Professional Workflows?, with 23 systems and 106 tasks. ② SaaSBench (arXiv 2605.17526) is from USTC and Amap (Alibaba), Exploring the Boundaries of Coding Agents in Long-Horizon Enterprise SaaS Engineering, with 30 tasks and 5,370 verification nodes. They are not the same thing, and search results frequently mix them up. An added irony: Alibaba led UniPat's Series A, and an Alibaba team then published a benchmark with the same name. The "enterprise SaaS coding evaluation" niche is being claimed by several parties at once, and UniPat has no exclusivity there.
Openness in detail (measured via the GitHub API on Sep 20, 2026): of UniPat's 9 public repos, only 4 carry an explicit licence: SaaS-Bench (Apache-2.0), RoadmapBench (MIT), EvoCodeBench (MIT) and harbor_multiturn (Apache-2.0). BabyVision, UniScientist, SWE-Vision, Terminal-X and unimath have public code but no licence file (strictly speaking not open source, and legally risky for enterprise use). Monthly-SWEBench has no public repo. Star counts are telling too: the coding benchmarks have only 16–28, while the models and BabyVision have 172–253. Things you must clone to run get very little attention; things you can download directly or that make headlines get more.
US · Academic / community standards28
| Benchmark | Size | Openness | Capability | Host | Cited by | Method |
|---|---|---|---|---|---|---|
| MMLU | 14,042 | Fully public | Knowledge across 57 subjects | UC Berkeley et al. | De facto universal standard | Pure API |
| MMLU-Pro | 12,032 | Fully public | Harder multi-task understanding | TIGER-Lab | Replacement after MMLU saturated | Pure API |
| GPQA Diamond | 198 (physics 82 / chemistry 78 / biology 38) | Public (some access limits) | PhD-level science problems | Rein et al. | Reported by OpenAI, Anthropic, Google and xAI | Pure API |
| BIG-Bench Hard | 23 tasks (>100 examples each) | Fully public | Semantic / numerical / logical reasoning | BIG-bench community | Partly replaced by GPQA/MMLU-Pro | Pure API |
| HLE | 2,500 (100+ subjects) | Fully public | Graduate-level frontier knowledge | CAIS + Scale AI | Adopted by major vendors as the high-difficulty standard | Pure API |
| GSM8K | 8.5K | Fully public | Grade-school math word problems | OpenAI | Saturated | Pure API |
| MATH | 12,500 | Fully public | High-school competition math | Dan Hendrycks et al. | Common standard | Pure API |
| AIME (as used for LLMs) | ~15 per year | Fully public | Top competition math | Mathematical Association of America | Standard for reasoning models | Pure API, needs multiple sampling |
| HumanEval | 164 | Fully public | Python function-level code generation | OpenAI | Highly saturated | Code sandbox (non-agentic) |
| MBPP | 974 | Fully public | Introductory Python programming | Google Research | Often paired with HumanEval | Code sandbox (non-agentic) |
| SWE-bench Verified | 500 (full set 2,000+) | Fully public | Fixing real GitHub issues | Princeton + Stanford + UChicago | Common standard for agentic coding | Docker; custom scaffolds allowed |
| LiveCodeBench | Rolling ~400–600 | Fully public | Competitive programming, rolling updates against contamination | UC Berkeley group | Often cited by vendors stressing decontamination | Code sandbox (non-agentic) |
| AgentBench | 8 environments | Fully public | Agent reasoning and decisions across environments | THUDM (Tsinghua) | Widely cited in academia | Unified text protocol |
| WebArena | 6 self-hosted sites / 812 tasks | Fully public | Real web navigation and operation | CMU | Recognised academic standard | Self-deployed sites; scaffolds not unified |
| τ-bench (incl. Airline) | Dozens to 100+ tasks | Fully public | Tool use + multi-turn customer-service interaction | Sierra | Listed on OpenRouter (123 models) | Officially separates standard and custom submissions |
| RULER | Four kinds of synthetic task | Fully public | Real effective context length | NVIDIA | Common long-context reference | Pure API |
| NIAH | A method, no fixed bank | Fully public | Long-text retrieval | Community (Greg Kamradt) | Saturated, replaced by RULER | Pure API |
| LongBench | 21 datasets (Chinese and English) | Fully public | Long-document QA, summarisation and 6 categories in all | Tsinghua KEG (Zhipu's predecessor) | Commonly cited for Chinese long context | Pure API |
| MMMU | 11.5K (30 subjects) | Fully public | Multimodal university-level understanding | Academic team | Multimodal standard | Pure API (image + text) |
| MathVista | 6,141 | Fully public | Math reasoning in visual contexts | AI4Math academic team | Commonly cited for multimodal math | Pure API (image + text) |
| Chatbot Arena / LMArena | No fixed bank; crowdsourced votes | Platform open source | Overall conversational preference | Started at LMSYS; spun out as a for-profit in 2026 | Once the most credible; declined amid vote-gaming disputes | Human voting platform |
| AlpacaEval | 805 fixed instructions | Fully public | Single-turn instruction-following preference | Stanford | Commonly cited in academia | API + LLM judge |
| MT-Bench | 80 multi-turn question sets | Fully public | Multi-turn conversation | LMSYS | Validation set for the LLM-as-judge method | API + judge |
| TruthfulQA | 817 | Fully public | Truthfulness against common misconceptions | Academic team | Common safety/truthfulness comparison | Pure API |
| SimpleQA | 4,326 | Fully public | Accuracy on short factual questions | OpenAI | Standard for factuality / hallucination rate | Pure API |
| BrowseComp | 1,266 | Fully public | Multi-step web browsing to locate information | OpenAI | Common for Deep Research-style products | Needs search tools; scaffolds not unified |
| DeepSearchQA | 900 (17 domains) | Open (HF: google/deepsearchqa, Apache 2.0) | Multi-hop deep-research QA with "causal chain" questions | Google DeepMind (not an independent academic team) | Listed on OpenRouter; few models tested | Needs search tool calls |
| WideSearch | 200 (100 Chinese + 100 English) | Open (GitHub + HF) | Breadth-first information gathering | ByteDance Seed team (not Western academia) | Few models tested | Needs search/browse tools |
Correction: on further checking, DeepSearchQA and WideSearch are not products of independent academic teams but in-house evaluations by Google DeepMind and ByteDance Seed respectively. Strictly, they belong in a fifth category this piece does not break out, "big-lab in-house", which is neither pure academic community nor a third-party vendor as defined here.
China · Academic / national-team standards7
| Benchmark | Size | Openness | Capability | Host | Cited by | Method |
|---|---|---|---|---|---|---|
| C-Eval | 13,948 (52 subjects) | Fully open | Knowledge and reasoning | Shanghai Jiao Tong + Tsinghua + Edinburgh | DeepSeek-V3, Qwen2.5 | Pure API |
| CLUE / CLUEWSC | 9 CLUE tasks | Fully open | Chinese NLU | CLUE academic community | DeepSeek-V3, Qwen2.5 | Pure API |
| BAAI FlagEval | 90+ datasets / 2M+ items | Open platform, partly open source | Language / multimodal image-text / text-to-image and text-to-video | Beijing Academy of AI (BAAI) | No formal citation found; BAAI publishes its own rankings | Some need environments |
| OpenCompass (Sinan) | 70+ datasets / 400K+ items | Fully open | Language / reasoning / knowledge / code / math / agents | Shanghai AI Laboratory | Explicitly cited in Qwen technical reports | Partly (agent dimension) |
| SuperCLUE | Three benchmarks, hundreds to thousands of items | Open | General + specialised + industry skills | SuperCLUE team (independent of CLUE) | Not cited in its own subjects' reports; a third-party tracking board | Needs interactive environment |
| ReLE (Feixian Zhineng) | 374 models / ~300 fine-grained dimensions | Open | Education / medicine / finance / law / agents | Feixian Zhineng (independent community) | None found; aimed at media and community | Partly (agent dimension) |
| CMMLU | 11,528 (67 subjects) | Fully open | Knowledge and reasoning in Chinese contexts | Independent academic team (published at ACL) | Explicitly cited by the Qwen series | Pure API |
Counting rules: 30 need a dedicated harness or environment (~47%), 3 partly need one, 24 are pure API, 4 need human review, 1 is a special platform (Chatbot Arena) and 2 have insufficient information. HumanEval, MBPP and LiveCodeBench do need an execution environment, but they are single-turn code generation plus sandbox tests with no multi-step agent scaffold, so the harness-sensitivity problem in the next section does not apply to them.
5. The harness trap: same model, different scaffold, up to 48 points apart
Harness and scaffold are not the same, but the industry uses them interchangeably
According to Hugging Face's agent glossary, the scaffold is the behaviour-definition layer: the system prompt, tool descriptions, how responses are parsed and what is remembered across steps, all settled before the first prompt. The harness is the execution layer: calling the model, dispatching tool calls and deciding when to stop, everything after the first prompt. The popular formula is Agent = Model + Harness. In practice the two are often treated as synonyms; Claude Code's official documentation calls itself "the agentic harness around Claude", folding scaffold duties into the harness. Like most papers, this piece calls both layers "harness".
Three layers: environment / agent (scaffold) / model
A benchmark run actually stacks three layers: the environment infrastructure (sandbox, Docker, deployable systems, and who maintains them) → the agent or scaffold (a standardised default, or bring-your-own, such as Terminus-2 vs Claude Code) → the model (the LLM actually being evaluated). The environment infrastructure we found splits into two ecosystems that do not interoperate:
The Harbor ecosystem
Open-sourced by the Laude Institute (creator of Terminal-Bench) under Apache 2.0. A unified task.toml format explicitly decouples the environment, agent and model layers. Built-in agents include claude-code, codex, aider, gemini-cli, opencode, OpenHands, goose and mini-swe-agent.
- BankerToolBench, ATLAS-Finance (Handshake)
- BiomniBench (Humanlaya; fixed harness = Terminus-2)
- Monthly-SWEBench, RoadmapBench, EvoCode-Bench (UniPat's Terminal-X suite)
- Terminal-Bench, SWE-Bench adapter layer
Each vendor's own stack
These also separate environment from agent, but design their own agent abstraction and do not share it.
- Scale: three-layer Docker images for SWE-bench Pro
- Mercor: Archipelago (in-house, for APEX-Agents; default ReAct toolbelt)
- HUD: hud-python SDK (OSWorld-Verified / SheetBench-50)
- Mechanize: own Docker (GBA Eval; one known submission used Claude Sonnet 5 + goose)
Hard data: score swings far exceed gaps in model ability
| Case | Fixed / standard harness | Native / custom harness | Gap |
|---|---|---|---|
| Claude Opus 4.7 · BiomniBench | Terminus-2: 63.94 | Claude Code: 73.34 | 9.4 pts |
| Claude Opus 4.5 · SWE-bench Pro | SEAL scaffold: 45.9% | Claude Code: 55.4% | ~10 pts |
| Similar models · SWE-bench Verified | mini-SWE-Agent | OpenHands: 77.6% | Several points |
| A model on WebArena | Original harness: 25% | Enhanced agentic scaffold: 55% | 30 pts |
| HAL statistics · SWE-bench Verified Mini | Top models across different scaffolds | Up to nearly 48 pts | |
The heart of the problem: "A leaderboard entry that reads 'model M, SWE-Bench Verified 65%' carries no information on its own, because you do not know what would happen with a different scaffold. Comparing two such numbers compares two systems, not two models." (From the paper Stop Comparing LLM Agents Without Disclosing the Harness, arXiv 2605.23950.)
Can vendors choose their own harness? Yes, and it is the default
- SWE-bench officially allows custom scaffolds (AutoCodeRover, Agentless, SWE-Agent and OpenHands all appear on the leaderboard). Each entry is really a "model + harness" system competing.
- Anthropic often reports scores with Claude Code directly. On Terminal-Bench 2.0, OpenAI is the only participant that insists on its own Codex CLI rather than the official Terminus-2.
- In Nov 2025 SWE-bench tightened its policy: Verified and Multilingual accept submissions only from academic institutions, and independent or commercial submissions are limited to the lower-bar Lite board. This is probably a response to commercial companies using custom harnesses to inflate scores, but Lite still allows custom harnesses.
- τ-bench is the only benchmark that institutionalises "standard vs custom submissions": replacing the default scaffold (adding a planner, extra agents, custom tools or a changed control flow) must be labelled separately.
Common open-source harnesses
| Harness | Category | Maintainer | Notes |
|---|---|---|---|
| SWE-agent | Code / SWE | Princeton-affiliated academic team | Introduced the agent-computer-interface paradigm; most cited |
| OpenHands | Code / SWE; general agent platform | Open-source community | Ships its own SWE-bench adapter; feature-heavy |
| mini-SWE-agent | Code / SWE | Open-source community | Minimal harness, often used as the standardised control |
| Terminus-2 | Terminal / CLI tasks | Terminal-Bench (Harbor) | Default standard harness for Terminal-Bench 2.0 |
| Agent-S / S3 | Desktop / computer use | Simular (open source) | First agent to beat humans on OSWorld (72.6%) |
| UI-TARS | Desktop / GUI grounding | ByteDance (open source) | Trained end to end; treats UI elements as first-class citizens |
| τ-bench harness | Customer service / tool use | Sierra (open source) | Simulated user + tools + database interaction |
| Harbor | General evaluation / RL environment framework | Laude Institute (open source, Apache 2.0) | Standardised infrastructure shared by several vendors |
| goose | General coding agent | Block (open source) | Used as the native harness for Mechanize's GBA Eval and others |
| OpenCode | General coding agent | Open-source community (MIT, 208k+ stars) | One of Harbor's built-in agents; supports 75+ model providers |
Harnesses the big model companies use for their own tests
| Company | Usual harness | Notes |
|---|---|---|
| Anthropic | Claude Code (own product) | About 10 points above the standard scaffold on SWE-bench Pro |
| OpenAI | Codex CLI (own product) | The only Terminal-Bench 2.0 participant not using the official Terminus-2 |
| Gemini CLI + an open-source computer-use-preview reference implementation | Mainly for browser/GUI; desktop operation officially "not yet optimised" | |
| Leading Chinese models (DeepSeek / Qwen / GLM / Kimi) | No public disclosure found | Agentic evaluations rarely state the scaffold; knowledge tests use the bare API |
The Holistic Agent Leaderboard (HAL) ran 21 controlled model-harness comparisons (Claude Code / Codex CLI / Pi, covering 7 models on SWE-bench Lite and Terminal-Bench 2.0). In 9 of 12 comparisons, switching to a harness that was not the vendor's own produced a higher score. A vendor's own harness is not necessarily the objectively strongest; it is more likely just the one it has tuned most carefully for itself.
6. AI for Science: no shortage of questions, but no shared standard
Nine in ten of the benchmarks on our list that carry a "science" label (GPQA, HLE, Riemann-bench, xbench-ScienceQA, MMMU, MathVista) measure scientific knowledge: whether a model understands science, not whether it can do research. Benchmarks that measure the process of AI doing research used to be extremely scarce, but that judgement was overturned in the first half of 2026 as a wave of new benchmarks appeared. The table below gives the overview; we then go deep on the heaviest of them, BiomniBench.
New research-agent benchmarks from H1 2026
| Benchmark | Released | Host | Size | What it tests | Open |
|---|---|---|---|---|---|
| AutoResearchBench | Apr 2026 (arXiv 2604.25256) | Academic team | – | Scientific literature discovery (Deep Research single-paper tracing + Wide Research batch retrieval); not a process evaluation | Open |
| ResearchClawBench | Jun 2026 (arXiv 2606.07591) | InternScience (Shanghai AI Laboratory / OpenGVLab) | 40 tasks / 10 scientific fields | Based on real published papers with the target paper hidden; expert multimodal rubric scoring. Same approach as BiomniBench but broader coverage | Open |
| SciAgentArena | Jun 2026 (arXiv 2606.12736) | Microsoft Research | ~200 tasks | Cross-scale research (single-cell omics, spatial omics, drug discovery, EHRs, genetics) with stepwise verification; 18 agents tested | Unclear |
| MLR-Bench | May 2025 (arXiv 2505.19955), accepted at NeurIPS 2025 | Academic team | 201 tasks (from NeurIPS/ICLR/ICML workshops) | The full idea → proposal → experiment → paper pipeline, auto-reviewed by MLR-Judge | Open |
| AIRS-Bench | Feb 2026 (arXiv 2602.06855) | Meta / Facebook Research | 20 tasks (from SOTA ML papers) | The whole research lifecycle (ideation → experimental analysis → iteration), with no baseline code provided | Open |
| HeurekaBench | ICLR 2026 | EPFL (mlbio-epfl) | Instantiated as sc-HeurekaBench (single-cell biology) | Really a framework for generating benchmarks: extract insights from papers → turn into QA → agent analyses and verifies autonomously | Open |
| StatefulDiscovery | Jun 2026 (arXiv 2606.11851) | Academic team | Reuses 40 real-data tasks | Strictly an evaluation of a method / agent architecture (explicit discovery-state control), not a standalone question bank | – |
Similar projects also found: InquiTree (arXiv 2606.09550, evaluating paper-derived "research tree" scientific inquiry), AstaBench (arXiv 2510.21652) and SoundnessBench (arXiv 2605.30329, testing whether AI can tell good research ideas from bad).
China's science evaluation landscape
China's science benchmarks cluster around the Shanghai AI Laboratory, plus projects involving HongShan's xbench and Humanlaya. They are characterised by systematic platforms more than single process-level evaluations:
| Benchmark | Host | What it tests | Level | AI performance |
|---|---|---|---|---|
| Sinan Scientific Intelligence Evaluation | Shanghai AI Laboratory (OpenCompass) | Evaluation platform covering the whole research workflow | Platform | Conclusion: clear weaknesses in models' scientific ability |
| SFE (Scientific Foundation Evaluation) | Shanghai AI Laboratory | 66 tasks, 5 scientific fields, bilingual; three levels: perception → attribute understanding → comparative reasoning | Knowledge / reasoning (multimodal) | Mainstream models score only ~30 |
| SciHorizon | Academic team (arXiv 2503.13503) | Tests 50+ LLMs on 16 dimensions across math, physics, chemistry, life sciences and earth/space; also assesses whether scientific data is "AI-ready" | Knowledge / reasoning + data readiness | Systematic diagnosis, no single headline score |
| SciEval | Academic team (NSFC-funded) | Multi-level science Q&A | Knowledge / reasoning | – |
| ResearchClawBench | InternScience (Shanghai AI Laboratory) | 40 tasks / 10 fields, hidden target paper, expert rubric | Research process | Best system (Claude Code) only 21.5/100 |
| xbench-ScienceQA | HongShan xbench | 100 science questions | Knowledge / reasoning | No third-party citations seen |
| BiomniBench | Stanford + Phylo + Humanlaya (a Chinese data lab took part) | See below | Research process | Frontier models 62–73 |
How well does AI do in science? Four judgements
- "Knowing science" and "doing research" are far apart. Multiple-choice GPQA is close to saturated, but once an agent must choose methods and interpret results itself, scores drop into the 20–70 range (ResearchClawBench 21.5, BiomniBench 62–73).
- The biggest problems are fabrication and shallowness. MLR-Bench found that 80% of cases fabricated or could not verify results; BiomniBench commonly sees key analysis steps skipped; SciAgentArena found agents could not produce novel insight. Work that looks complete but does not hold up scientifically is more dangerous than a wrong answer.
- AI has roughly caught up with junior scientists but not senior ones. BiomniBench's early human baseline: senior pharma scientist 68.5, best AI 65.0, junior scientist 48.5.
- The shared bottleneck in the US and China is the lack of a consensus standard. The US is strong in heavily resourced process-level evaluations from several institutions (Meta, Microsoft, Stanford, FutureHouse); China is strong in systematic platforms (Sinan, SFE, SciHorizon). Neither side has a research standard that several leading labs have written into their system cards.
AutoResearchBench's Deep Research accuracy is only 9.39%. The best system on ResearchClawBench (Claude Code) averages only 21.5/100. MLR-Bench found coding agents fabricated or could not verify experimental results in 80% of cases. On AIRS-Bench, agents beat the human SOTA on only 4 of 20 tasks. More importantly, we found no case of any of these benchmarks being formally cited in a major model company's system card. Academic analysis states plainly that "research-agent benchmarks remain fragmented, covering different slices of the research lifecycle (ideation/planning vs execution) that do not connect."
A harness-sensitivity case more extreme than BiomniBench: GAIA
GAIA is a general agent benchmark (not research-specific), but in May 2026 it produced a startling data point: on the same 466 questions, scores on different leaderboards ranged from 44.8% to 92.36%. That gap is not measurement error; it comes down to which tools or scaffold were allowed. It is even more extreme than the 10–48-point harness gaps for SWE-bench and BiomniBench in §5, and the logic is the same: research-type tasks are more open-ended and their rubrics more subjective, so harness sensitivity will only be worse than for coding benchmarks, not milder.
BiomniBench: still the heaviest example (no longer the only one)
The new benchmarks above show that research-agent evaluation has gone from empty to crowded, but BiomniBench remains the most heavily resourced of them: 100+ senior experts, over 1,000 hours of development, a direct companion to Stanford's production research agent Biomni, and a six-dimension rubric plus a two-track design (fixed harness vs native harness) that is the most complete among these new projects. The details:
- Scale: 100 data-analysis tasks (50 public + 50 private against contamination), 17 task types, 5 disease areas plus general biology.
- Source of questions: each is based on a real paper published in Nature, Cell or Science, developed with the original authors or domain experts.
- Effort: 100+ senior experts and over 1,000 hours from Stanford, Harvard, Peking University and several leading pharma companies, together with Phylo and Humanlaya Data Lab.
- Companion to: Biomni, the general biomedical AI agent from Stanford's SNAP lab (105 software packages + 59 databases + 150 specialised tools).
- Core innovation: process-level evaluation. The agent's full trajectory is scored against an expert rubric, not just the final answer.
Scoring dimensions and examples
| Dimension | Core check | Example criterion |
|---|---|---|
| Data handling | Loads and prepares data correctly | Loads the required files, merges on the right identifiers, filters to the right treatment group |
| Method choice | Chooses an appropriate analysis method | For survival analysis, uses Kaplan-Meier curves + log-rank test |
| Statistical rigour | Applies statistical tests correctly | Applies multiple-testing correction (BH-FDR) |
| Source reliability | Relies on trustworthy references | Provides relevant citations from peer-reviewed literature |
| Scientific reasoning | Logical consistency and biological connection | Avoids overclaiming causation; links findings to real mechanisms |
| Biological interpretation | Sixth dimension added in the final version | – |
Typical failure modes: skipping the MYC gene-specific analysis, skipping the phenotype-correlation analysis, skipping the key pathway interpretation. The tasks are not "run a script and output a number"; the agent has to know which direction to dig.
Which harness is used
Underneath, the Harbor execution framework standardises the environment (task.toml configures verifiers and environment budgets), so data, tools and sandbox resources are identical across agents. Grading is not an automated verifier but an LLM judge. The paper compared how well 5 candidate judges (Claude Opus 4.6, DeepSeek V3.2, Gemini 3.1 Pro, GPT-5.4, Qwen 3.6) agreed with internal experts A/B/C and chose Gemini 3.1 Pro as the judge throughout (closest to the human experts).
How the models scored
Official fixed-harness comparison: with every model on Terminus-2, the top models were within 1.6 points of each other, clustered at 62–64. Official native-harness comparison (each vendor's own tuned agent):
| Combination | Score |
|---|---|
| Claude Code + Opus 4.7 | 73.34 (highest) |
| Claude Code + Opus 4.6 | 69.51 |
| Codex CLI + GPT-5.4 | 68.69 |
| Codex CLI + GPT-5.5 | 67.59 |
| Claude Code + Sonnet 4.6 | 66.39 |
| Terminus-2 + Opus 4.7 (fixed-harness control) | 63.94 |
Early-version human baseline (v0, 15-task subset, published on Phylo's blog):
| Rank | System or person | Score |
|---|---|---|
| 1 | Senior pharma scientist (5+ years) | 68.5 |
| 2 | Biomni Lab | 65.0 |
| 3 | Edison Analysis | 52.7 |
| 4 | OpenAI Agents SDK (GPT-5.2) | 51.4 |
| 5 | Junior scientist (3 years) | 48.5 |
| 6 | Claude Code (Opus 4.6) | 47.8 |
At the time, the best AI system (65) was close to or above the junior scientist (48.5) but clearly below the senior scientist (68.5). Direct "AI vs real expert" comparisons like this are rare.
Independent third-party replication (the omicverse team ran its own agent, "omicos", with different base models on the 50 public tasks): GPT-5.5 78.8%, DeepSeek4-pro 73.0%, GPT-5.4 67.7%, DeepSeek4-flash 66.0%, MiMo-v2.5-pro 65.8%, MiMo-v2.5 62.3%, GPT-5.4-mini 44.2%. A counter-intuitive finding: GPT-5.5 had the highest total but only 60% on biological interpretation, while some cheaper models (DeepSeek4-flash 76%, MiMo-v2.5-pro 73%) beat it on that dimension. Models generally cite their sources honestly, but consistently fall short on method choice and biological interpretation, the two dimensions that need real scientific judgement.
FutureHouse's LAB-Bench (2,457 questions, 8 categories and 30 subtasks, testing basic biology research skills including protocol troubleshooting in ProtocolQA and molecular-cloning scenarios; Anthropic reports scores on it) is another process/task-level evaluation outside the table above. Periodic Labs' "lab as reward function" (the model reads literature → runs quantum-chemistry calculations → an automated lab synthesises → measured results feed back as training signal) is a training paradigm rather than a benchmark, and sidesteps evaluation fragmentation altogether; for the full breakdown, and why the three big labs cannot do it, see §7. Anthropic × Benchling (research software connected directly) likewise bypasses the third-party benchmark layer.
7. Rubric grading: turning "unverifiable" into "verifiable", at the cost of swapping "right" for "complete"
The last section concluded that research benchmarks are crowded but lack consensus. Look only at method, though, and the US and China have converged on the same solution: break an open-ended task into N small, independently judgeable rubric items, then aggregate them into a score or reward. This section covers three things: how the mechanism works, how far it is from a true standard, and which methods have been shown to narrow the gap.
Rubrics make open-ended tasks gradable, but they reward "mentioned it", not "got it right", and they do not penalise "said too much". In medical Q&A that substitution costs little; in research it can be fatal. A research report that mentions everything but reaches the wrong conclusion will get a high rubric score.
Not new: a timeline of the paradigm
The standard names are rubric-based reward or rubrics as rewards. The motivation is that RLVR (reinforcement learning from verifiable rewards) has been hugely successful in math and code but cannot be used for tasks without a single answer, such as medical or scientific writing. The underlying assumption, that "verifying is cognitively much cheaper than creating from scratch", comes from the older scalable-oversight line of work (CriticGPT, debate-based alignment) and is not original to any new company.
| Date | Work | Key contribution |
|---|---|---|
| 2024 | OpenAI Rule-Based Rewards / CriticGPT | Deriving rewards from rule text; model-assisted human verification |
| May 2025 | OpenAI HealthBench | 262 physicians (practising in 60 countries), 5,000 conversations, 48,562 physician-written rubric items, weighted, graded by GPT-4.1 |
| Jul 2025 | Scale AI Rubrics as Rewards (RaR), arXiv 2507.17746 | First systematic framework: four weighted item categories + GRPO; released RaR-Medical-20k / RaR-Science-20k |
| Aug 2025 | Xiaohongshu RL with Rubric Anchors, arXiv 2508.12790 | The largest rubric reward system at the time, 10,000+ items, explicitly sourced through human-machine collaboration; open-sourced Qwen-30B-A3B |
| Sep 2025 | Chasing the Tail, arXiv 2509.21500 | Theory locating the root of reward over-optimisation in high-score tail mismatch |
| 2026 | Open Rubric System / Autorubric / OpenRubrics / ARBOR / SibylSense / RUC-NLPIR Rubrics Survey | Signs of a mature paradigm: surveys appear, along with papers dedicated to failure modes |
A direct implication for §8: UniPat's blog claims that "the two existing paradigms (purely human / purely synthetic) do not exploit this asymmetry." That is inaccurate against the literature. Scale's RaR came more than six months earlier, and Xiaohongshu's Rubric Anchors seven months earlier, explicitly describing human-machine collaboration. UniPat's real differentiation lies in domain coverage (50+ disciplines) and expert time per sample (1–2 hours), not in method.
What a rubric item actually looks like (examples from the RaR paper appendix)
Each question gets 7–20 rubric items, each with a title (2–4 words), a description (one sentence, starting with its category) and a weight. Items must be self-contained enough that "a non-expert can judge from this item alone, without inference or external sources". That is the precondition for an LLM judge to act as grader.
| Category | Weight | Meaning | Example from the paper (medical question) |
|---|---|---|---|
| Essential | 5 | Key fact or safety check; if missing, the whole answer is invalid | Essential Criteria: Identifies non-contrast helical CT scan as the most sensitive modality for ureteric stones. |
| Important | 3–4 | Key reasoning, completeness, clarity | Important Criteria: Explains that non-contrast helical CT detects stones of varying sizes and compositions. |
| Optional | 1–2 | Style or extra depth | Optional Criteria: States "The final answer is (B)" or similar answer choice formatting. |
| Pitfall | −1 / −2 | Common errors or omissions for this question; negative points | Pitfall Criteria: Does not mention identifying (B) as the correct answer. |
RaR's experiments on Qwen2.5-7B produced two counter-intuitive results. Implicit aggregation (handing the whole rubric to the judge for one overall score) beat explicit aggregation (judging item by item and summing weights), contrary to the intuition behind atomic verifiability. And in the science domain (GPQA-Diamond), RaR-Explicit did worse than the Reference-Likert baseline (0.3030 vs 0.3775). The benefit of rubrics depends on whether the task is truly open-ended; for multiple-choice tasks, decomposition adds little.
Five mechanisms research benchmarks use for rubrics
A · Weighted rubric tree (most fine-grained)
PaperBench (OpenAI, arXiv 2504.01848): replicating 20 ICML 2024 papers. The rubric is a requirements tree whose leaves are single pass/fail conditions, with each node hand-weighted relative to its siblings, 8,316 leaf nodes in all. The decomposition rule is strict: keep splitting until an expert can judge a node within 15 minutes, turning atomicity into an operational cost definition. Rubrics were co-written with each paper's original authors; grading uses o3-mini as judge, and a separate JudgeEval was built specifically to evaluate the grader.
B · Fixed-point checklist + pass mark (simplest)
FrontierScience-Research (OpenAI, arXiv 2601.21165): 60 original research subtasks designed by PhD scientists (doctoral students, professors, postdocs). Each has a 10-point rubric of independent, objectively judgeable items, each with a pass/fail description and points (for example "writes out equation X"); ≥7/10 counts as success. It drops exact matching in favour of evaluating intermediate reasoning steps. It is also the leaderboard UniPat's UniScientist reports on.
C · Dimension system + dynamic weights (for open-ended reports)
DeepResearch Bench (arXiv 2506.11763, ICLR 2026) uses the RACE framework: four orthogonal dimensions (comprehensiveness, insight depth, instruction following, readability); the judge generates task-specific weights per question (averaged over T trials) and then scores relative to a high-quality reference report. DEER (arXiv 2512.17776, ICML 2026): an expert taxonomy of 7 dimensions / 25 sub-dimensions → 101 fine-grained rubric items, plus a claim-verification architecture that checks both cited and uncited assertions. BiomniBench's six-dimension process-level evaluation belongs here too (see §6).
D · Atomic fact decomposition (no pre-written rubric)
HeurekaBench (ICLR 2026, arXiv 2601.01678): both the agent's answer and the ground truth are decomposed into atomic facts and compared for presence and correctness. GPT-4o grading this way reaches Spearman ≈ 0.90 against PhD-level experts. It is currently the only design that closes the "only checks whether it was mentioned" loophole at the mechanism level, because it also requires correctness.
E · Rubric + hard gates + real experiments (the only link to the physical world)
BioLab Bench: task-specific metrics + rubric verifiers + equipment-level validity gates + real experimental tests, chained in four layers. A cheaper alternative is BenchBench-Protocol (arXiv 2608.23898): 149 tasks drawn from 96 published protocols on Benchling, where the difference between the published version and the scientist-modified version is the ground truth, using completed wet-lab experiments as delayed ground truth.
The anti-LLM camp · an unresolved split
ResearchRubrics (Scale × UChicago, arXiv 2511.07685): 101 prompts / 2,593 human-written items, explicitly forbidding LLMs from generating or seeding any item, on the grounds that "having humans only verify LLM-generated rubrics introduces anchoring bias". Triple review and 2,800+ human hours. This directly conflicts with the "LLM generates, humans verify" route of UniPat and HeurekaBench, and nobody has run a head-to-head experiment on the same questions.
| Benchmark | Scale | Who writes the rubric | Mechanism | Grader reliability published? |
|---|---|---|---|---|
| PaperBench | 20 papers / 8,316 leaves | Co-written with original authors | Weighted requirements tree, 15-minute rule | Yes, JudgeEval evaluates the grader |
| FrontierScience-Research | 60 subtasks | PhD scientists | Independent 10-point items, ≥7 to pass | None found |
| ResearchRubrics | 101 prompts / 2,593 items | Human only; LLM generation banned | Multi-axis + triple review, 2,800+ human hours | Relies on triple review |
| DeepResearch Bench | 100 tasks / 22 fields | Generated dynamically by the judge | RACE: 4 dimensions + dynamic weights + relative scoring | PAR 71.33% vs human inter-rater 68.44% |
| DEER | 50 tasks / 101 items | Expert taxonomy | 7 dimensions, 25 sub-dimensions + claim verification | Task-specific expert guidance |
| BiomniBench-DA | 100 tasks | Original authors or domain experts | Six-dimension process level, whole trajectory | Compared 5 candidate judges for agreement (see §6) |
| HeurekaBench | OEQ + MCQ | LLM-generated, verified against published conclusions | Atomic facts: presence + correctness | Spearman ≈ 0.90 |
| SciAgentArena | ~200 tasks | Not examined | Stepwise verification | None found |
| BioLab Bench | Not disclosed | Wet-lab experts | Rubric + equipment gates + real experiments | Ground truth from experiments |
| HealthBench (medical; origin of the method) | 5,000 conversations / 48,562 items | 262 physicians | Weighted items, graded by GPT-4.1 | Agreement with physicians ≈ physician inter-rater agreement |
When you look at any rubric leaderboard, the first question is not "who is on top" but "how well does your grader agree with human experts?" Only four of the ten in the table give a number.
How far from a true standard: two gaps, and the second is structural
The breakdown comes from Reward Hacking in Rubric-Based RL (arXiv 2605.12474): the policy optimises against a training grader but is evaluated by a panel of three frontier models from different families.
Gap one: grader failure (controllable). The training grader says an item is satisfied; the reference graders say it is not. Weak graders produce large spurious gains that do not transfer, and exploitation grows as training proceeds, concentrated in three recurring failures: partially satisfied compound conditions counted as passes, implied content treated as explicit statements, and loose topic matching. A stronger grader reduces this sharply but cannot eliminate it.
Gap two: the limits of rubric design itself (structural). Even with a very strong rubric grader, the responses it prefers look worse to a judge who ignores the rubric and assesses overall quality:
| Change after rubric RL | Direction |
|---|---|
| Completeness and "mentioned X" presence items | Up |
| Factual correctness | Down |
| Concision | Down |
| Relevance | Down |
| Overall quality | Down |
Rubric graders prefer the post-RL checkpoint, while rubric-free judges prefer the pre-training base model. The rubric score went up and the model got worse. The mechanism is easy to see: a rubric is a checklist, so the model's optimal strategy is to mention every item, producing output that is long, exhaustive and verbose, with lower factual density. The paper concludes that stronger verification reduces reward hacking but does not by itself guarantee that rubric gains correspond to broader quality improvements.
HealthBench's meta-evaluation (physicians re-reviewing 34 consensus items, criterion-level Macro-F1) shows that pairwise agreement between the model grader and physicians matches agreement among physicians themselves. Put the two results together for the full picture: rubric graders already judge single items at human level; the problem is the aggregation assumption that adding up N items equals quality. The gap is not precision but metric structure.
Methods shown to narrow the gap (ordered by strength of evidence)
- Cross-family grader panels. Strictly separate the training grader from the evaluation grader; the hard rule is never let the generating model and the grading model come from the same family. Autorubric and PoLL-style ensembles show that when variance and threshold reliability matter, a jury of diverse judges beats a single strong model.
- Human-label calibration plus periodic recalibration (the most mature in engineering). Hand-label 100–300 representative outputs with the same rubric the grader will use, compute Cohen's kappa, Pearson and MAE, and target kappa > 0.6. Recalibrate quarterly to catch drift, and always recalibrate after the grading model is upgraded.
- Pairwise comparison instead of pointwise scoring. Open Rubric System (arXiv 2602.14069) uses pairwise adaptive rubrics and tournament ranking to cure "discriminative collapse" (pointwise scoring squeezes all responses into the same band), and requires running both orderings to remove position bias.
- Focus on the high-score tail. Chasing the Tail (arXiv 2509.21500): over-optimisation stems from being unable to tell "excellent" from "merely very good". The value of rubrics is that they can use off-policy samples while being insensitive to their surface artefacts; the key is giving rubrics the power to discriminate among several very good responses.
- Induce rubrics backwards from failed trajectories. The DeepVerifier approach keeps rubrics close to the real distribution of failures rather than the failures experts imagine.
- The self-internalization gap. A grader-free diagnostic based on the policy model's own log-probabilities that can detect when a policy trained with a weak grader stops genuinely improving. A low-cost early warning.
- Explicit negative items / suppressing the presence bias. Add Pitfall items, down-weight "mentioned it" items, up-weight "got it right" items, add concision constraints. The logic fits the diagnosis, but no paper has validated it systematically; it is an inference.
Specialised models are taking over the grading layer: TypeSafe Jev
Rubric grading is essentially "answer N independent closed questions about one output", which is exactly the native form of TypeSafe Jev (early access since Sep 15, 2026; currently jev-1.13.0). It takes unstructured state as input and outputs typed decisions with calibrated probabilities in one parallel pass, without generating any text. Its primitives bool / score / choice map to a rubric's met/not met, graded items and failure-mode classification. Criteria are written in natural language and need no labelled data. Pricing is $0.042/MTok input with free output, and latency is 70–500 ms. In Good Start Labs' test of 6,003 checks, Jev cost about $160 per million judgements versus about $33,000 for Claude Fable at a comparable level (GPT-5.6 Luna about $400, Gemini 3.8 Flash about $1,600). That enables a real architectural change: what used to be too expensive to do except by sampling can now be done on every single trajectory.
An independent calibration audit (8,000 judgements × 4 phrasings × 2 base rates, compared with human labels rather than model opinions) found that Jev ranks well (AUC 0.90–0.91), but its probabilities are systematically biased toward "yes", and the rarer the positives, the worse the bias. When Jev reports about 75% confidence, only 10% of human labellers would flag the item. Raw ECE ranges from 0.157 (19.2% positive rate) to 0.209 (2.9% positive rate); a two-parameter monotonic recalibration brings ECE down to 0.023/0.007 with AUC unchanged. The same audit yields a by-product especially useful for rubric work: running four phrasings of the same question on identical state moved the mean probability from 0.445 to 0.566 and ECE from 0.397 to 0.519, while AUC barely moved. Jev can therefore expose how fragile a rubric item's wording is at very low cost.
Three more hard constraints settle its place in research evaluation: it gives no reasons (you cannot learn why an item was not met), it cannot abstain (there is no "insufficient information" state), and its context is only 64K total / 32K state, with context rot. The last rules out the most important direction from §6, process-level evaluation: BiomniBench must assess a full agent trajectory, PaperBench a whole replication artefact, DEER a long report, and none fits in 32K. Note too that the vendor's own benchmark uses the average opinion of GPT-6 Astra and Fable 5.1 as the reference answer, not human ground truth, and its tasks were written by its own team.
So its proper place is: the execution layer for leaf-level judgements (PaperBench's "judgeable in 15 minutes" rule defines exactly this kind of short, self-contained, atomic question), large-scale exhaustive scanning, routing by confidence to humans, and the architecturally different member of a cross-family grader panel. It is not suited to being the sole judge, grading trajectories or diagnosis. One judgement must be stated plainly: Jev cuts grading cost by two orders of magnitude, but doing the wrong thing cheaply is still wrong. Its probabilities lean systematically toward yes, and the known disease of rubrics is precisely inflated presence items. Wiring it uncalibrated into an RL reward loop pours fuel on reward hacking.
The ground-truth layer: how wet-lab experiments connect
A rubric can only judge whether a piece of research looks like good research, not whether its conclusion is right. In science only the latter counts. So the real question is whether the one ground truth that cannot be Goodharted, physical experiment, can enter the training loop. The answer: all three big labs use wet-lab work, but none publicly uses it as an RL reward signal.
| Tier | Approach | Who |
|---|---|---|
| Tier 1 · Post-hoc validation only | Wet-lab results are launch evidence, not part of the training loop | OpenAI × Retro Biosciences (GPT-4b micro designed RetroSOX/RetroKLF, raising pluripotency-marker expression by more than 50×; over 30% of SOX2 suggestions beat wild type; iPSC generation cut from 3 weeks to 7 days; model not publicly released). Anthropic, Aug 2026: de novo protein binder design against 15 targets with a hit rate of 22–35% (industry norm 10–15%), validated by Adaptyv Bio + Twist. DeepMind: AI co-scientist in real labs |
| Tier 2 · Results fed back as training data | AI proposes hypotheses and molecules → humans test at the bench → results flow back to improve the model (the wording is "improve the model", not "use as RL reward") | Anthropic has gone furthest: its own wet lab in the San Francisco Bay Area with hired bench scientists; a Life Sciences Verification Program offering up to USD 1M in Claude credits plus funding for external wet-lab validation (Modal contributing USD 250K of compute, Twist providing DNA synthesis). In essence, crowdsourced ground-truth data |
| Tier 3 · Wet lab as RL environment | Physical experiments serve as the RL environment, with reality providing feedback | Only new companies: Periodic Labs (founded by Liam Fedus, former head of post-training at OpenAI, and Ekin Dogus Cubuk, a former DeepMind GNoME author), with a 1-trillion-parameter MoE model Neon trained on a peak of 1,300 H200s; "a scientific rollout can reason and run tools for over an hour, while a training step takes only minutes", so training and inference run asynchronously. Lila Sciences' AI Science Factories are similar |
Why the big labs do not do this: four structural reasons. ① The time scales differ by 3–5 orders of magnitude (training steps take minutes, experiments take days to weeks, and one RL run needs hundreds of thousands of rollouts). ② Sample efficiency and cost do not match (feeding the most expensive signal to the hungriest algorithm). ③ The signal is sparse and noisy, making credit assignment nearly impossible (a failure might come from a wrong hypothesis, or from pipetting or a reagent batch). ④ The trade-off between general and vertical models: wet-lab signal is only valid in very narrow domains, and Periodic and Lila dare to do it precisely because they gave up generality.
So frontier models actually rely on four substitutes for wet-lab work: formal verifiers (Lean, seconds, but only for math), computational simulation proxies (DFT, molecular dynamics, docking; minutes to hours, with systematic deviation from reality), retrospective ground truth (completed experiments the model has not seen; zero cost but contamination risk) and rubrics (seconds, but with the structural bias described in this section). Wet-lab work itself serves as final endorsement and calibration, not as a source of gradients.
Research benchmarks with wet-lab ground truth are necessarily small, slow and expensive (BioLab Bench and BiomniBench are both in the hundred-task range) and can never reach the scale of HLE, so their role should not be a leaderboard but a calibrator. The future of research evaluation is not one benchmark but two layers: a large, cheap rubric layer for day-to-day iteration, and a small, expensive ground-truth layer that periodically calibrates the first. Whoever controls the second defines how credible the first is. Today almost everything in both the US and China is the first layer, and nobody builds the calibration layer in between (regressing rubric weights on ground-truth outcomes). UniScientist uses uniform weights, RaR uses hand-picked 5 / 3–4 / 1–2 tiers, and HealthBench uses physicians' importance judgements; none of the three calibrates against outcomes.
What to watch: when Periodic Labs reports its first new material or superconductivity result obtained through a closed RL loop (rather than human intuition plus AI assistance). That is the only hard test of whether the Tier 3 route works. Its Sep 2026 technical post covered only training and serving infrastructure and explicitly reported no newly validated materials or superconductivity results.
8. China's newcomers: UniPat, Humanlaya and Zhiyan Huisheng
China's "AI evaluation plus high-quality training data" companies fall roughly into three tiers. The sector only really got going in the second half of 2025, and most of the leaders are less than a year old. The first tier is the true counterpart to Mercor, AfterQuery and Turing in the US.
| Date | Event | Detail |
|---|---|---|
| Late 2025 | UniPat AI founded | Founder Kuan Li (Li Kuan), formerly of Alibaba's Tongyi AI Lab; focus on post-training analysis, data synthesis and RL |
| 2025 | Humanlaya (formerly Rang Zhineng) founded | Haidian, Beijing; an AI data lab bringing together PhDs and research experts |
| Jan 2026 | Zhiyan Huisheng founded | Legal representative Zhu Xizhou; very little public information |
| Jun 2026 | Tencent (via Shanghai Qishan Investment) invests in Zhiyan Huisheng | – |
| Early Sep 2026 | Humanlaya closes a Pre-A round of several hundred million RMB | Led by CDH Investments (Hong Kong fund), with HongShan, Capital Today and BAI Capital participating |
| Sep 2026 | UniPat AI closes a USD 300M Series A | Led by Alibaba, with Tencent and HongShan participating; valued at about USD 2.5B, the largest deal in the sector |
The tiers
Tier 1: new-generation evaluation and expert-data companies
- UniPat AI: coding, browser and multimodal visual-reasoning evaluations; the Terminal-X suite (DeepTerminalBench / EvoCode-Bench / RoadmapBench, all built on the Terminal-Bench 2.0 format + Harbor) plus Monthly-SWEBench, SaaS-Bench and ClawBench; and it trains its own models (UniScientist-30B-A3B, UniMath-35B-A3B), see below
- Humanlaya: One-Million Bench / BiomniBench; high-end expert annotation across medicine, finance, law, manufacturing and education
- Zhiyan Huisheng: AI industry application systems integration; business details unverified
Tier 2: traditional annotation firms upgrading
- Haitian Ruisheng (688787.SH): listed on the STAR Market; 2025 revenue RMB 377M (+59%)
- Datatang, Testin Data, DataBaker, Zhengshu Zhineng, TalkingData
UniPat, Humanlaya and Zhiyan Huisheng all have HongShan involvement, on top of HongShan's own incubated xbench evaluation system (two tracks: a capability-ceiling track and a real-world-value track; "evergreen evaluation" updated dynamically; it has released xbench-ScienceQA, DeepSearch and AgentIF, and BabyVision jointly with UniPat). HongShan is the most densely positioned investor in the "evaluation defines model value" sector.
UniPat is not just an evaluation company: it trains its own models (added Sep 20, 2026)
Earlier versions missed this, and it changes how the company should be characterised. Besides the benchmarks catalogued in §4, UniPat's GitHub organisation (UniPat-AI, 9 public repos) holds three non-benchmark products:
| Product | Spec | Method | Self-reported results |
|---|---|---|---|
| UniScientist-30B-A3B Mar 2026, research-agent model | Qwen3-MoE architecture, 30B total / 3B active, Apache-2.0; blog only, paper marked "Coming Soon" | Evolving Polymathic Synthesis: an LLM acts as cross-disciplinary question writer and human experts as high-precision verifiers; rubrics pass three filters (objective consistency, discriminability, atomicity). Training = agentic SFT + a Report Aggregation objective (synthesising a final report from N candidates, with references produced by rubric-based rejection sampling). Dataset: 4,700+ research-grade samples, 20+ rubric items per question, 50+ disciplines, 1–2 hours of expert time per sample | FrontierScience-Research 28.3 (33.3 at Aggr@8) vs GPT-5.2 25.2 / Claude Opus 4.5 17.5; DeepResearch Bench II 48.0 vs OpenAI DR 45.4. Full reasoning trajectories released (far more responsible than releasing scores alone); training data not released |
| UniMath-35B-A3B Jul 2026, olympiad-level proof model | Base explicitly Qwen/Qwen3.6-35B-A3B, 35B total / 3B active, Apache-2.0; also blog only | Training data supervises the revision process of a proof: initial draft (exposing obligations to prove) → self-assessment (a "proof debt ledger" naming the first real obstacle) → targeted repair or a change of route → final synthesis. At inference, pool_refine_tournament: sample, self-assess and rank → repair the strongest imperfect candidate → merge survivors pairwise, keeping only measurable improvements | IMO 2025 35/42, USAMO 2026 36/42, IMO-ProofBench 86.0% (without TTS: 19.75 / 17.88 / 52.6%) |
| SWE-Vision | Agentic VLM framework (stateful Jupyter inside Docker) | A tool, not a benchmark | – |
① The with-TTS and without-TTS scores use different grading. The former is avg@8 graded by a single judge, DeepSeek-V4 Pro; the latter is the final proof graded by three judges (Gemini 3.1 Pro + GPT-5.5 (xhigh) + DeepSeek-V4 Pro) plus human expert review. How much of the jump from 19.75 to 35 comes from TTS and how much from the change in grading cannot be separated from public information, which is once again the theme of §5. ② "Gold medal" is relative to the IMO cut-off, not to SOTA. On IMO-ProofBench it ranks fourth (GPT-5.6-Sol at 91.1% and Claude Fable 5 at 89.0% are both above it). The real story is lifting a base model from 23.1% to 86.0%: an impressive gain, but still below frontier closed models. Also, the blog text says "IMO 2026" while the table says IMO 2025; when citing, go by the table.
Team lineage: UniScientist is essentially a sequel to Tongyi DeepResearch
UniScientist's citation lists 9 authors with affiliations 1 UniPat AI, 2 Peking University (no Alibaba affiliation). But Kuan Li is the founder, and Baixuan Li and Jialong Wu were, with him, core authors of the Tongyi DeepResearch technical report and WebSailor (Tongyi Lab, Alibaba Group). The methodological lineage is plain: long-horizon information-seeking agents, tool loops and rollout aggregation all extend the WebSailor / Tongyi DeepResearch line.
Two easily confused points need stating clearly. ① The two models are not "made by Tongyi"; they are built on open Qwen weights (UniMath's HF metadata states its base_model explicitly; UniScientist's architecture tag is qwen3_moe, and the only community discussion on its HF page is one reminding it that it does not name its base). ② Whether all these authors are on UniPat's payroll cannot be confirmed; some may still be at Tongyi, in a cross-institution collaboration.
But this relationship explains why Alibaba led a USD 300M round. Chinese media described it as "Alibaba to lead investment in former employee's AI evaluation company", and a Sina headline was blunter: "backing a former intern's startup". For Alibaba it achieves three things at once: retaining talent, free ecosystem endorsement (all of UniPat's models are built on Qwen), and locking in an evaluation-data supplier.
The question-setter is also a contestant, and the referee's technical taste has an origin. UniPat is described as a third-party evaluator, but its base models come from Alibaba, its money comes from Alibaba and its team comes from Alibaba, and its leaderboards are grading Qwen 3.6 Plus and its competitors. No cheating is needed for bias: a research or agent evaluation designed by people from Tongyi DeepResearch will naturally reward that style of solution (long-horizon search, tool loops, rollout aggregation), which is exactly what Qwen-family models are good at. This is a different kind of problem from the US pattern of "customers citing their own supplier", and harder to spot.
It is not yet a closed loop: its own benchmarks cover coding, GUI and vision, while its models target research and math, so the two do not overlap. The real risk point is the day it uses the same Evolving Polymathic Synthesis pipeline to produce a research benchmark. Training and test sets would then be products of the same distribution. That is not cheating, but the evaluation would lose its information value.
Hugging Face downloads: UniScientist 75 (since its March release), UniMath 18 (since July), with 15 and 2 likes; 1 and 0 HF discussion threads, and that single thread is the one pointing out the missing base model. No posts found on Reddit (including r/LocalLLaMA) or Hacker News. There is Chinese coverage, but the same article, "Can AI really do research…", appeared in Sina Finance's "company news" section, PingWest, TMTPost and a Zhihu column, which is PR distribution; the only genuinely independent editorial coverage found was from QbitAI.
A 30B / 3B-active model is the size the local-deployment community should love most, yet nobody runs it: no GGUF quantisation, no hands-on posts. The reasonable characterisation is a "fundraising narrative asset" rather than an "open-source model", serving as technical backing for a USD 2.5B valuation. As a business that makes complete sense: UniPat's customers are model makers and enterprises, 75 downloads do not affect its data sales, and one QbitAI article may be worth more than ten thousand downloads.
Team background checks
UniPat's founder is public (Kuan Li, formerly of Alibaba's Tongyi AI Lab); it is the only one of the three that actively discloses its founder and uses big-tech technical background as endorsement. The founder of Humanlaya (Rang Zhineng) has not been named in any public report; the website has no team page and the GitHub organisation's member list is private, so only the investor line-up can be confirmed. For Zhiyan Huisheng, public business-registry records show only the legal representative; there is no public reporting on team or product, making it the lowest-profile of the three.
Citation: the "external adoption" step has not happened yet
We found no evidence that UniPat, xbench or Humanlaya benchmarks are formally cited in the official technical reports of DeepSeek, Qwen, Zhipu GLM, Kimi, MiniMax or other leading model makers. Current "citations" happen mainly among insiders (UniPat and xbench jointly releasing BabyVision is essentially one ecosystem endorsing itself). By contrast, C-Eval, CLUEWSC, CMMLU and OpenCompass are formally cited in the DeepSeek-V3 and Qwen technical reports. These new companies' credibility currently shows up in funding markets (Alibaba and Tencent leading, a USD 2.5B valuation) rather than in technical-report citations, two entirely different states of trust.
A recheck on Sep 20, 2026 confirmed this and allows a finer grading. Only four kinds of "citation" turned up, and the first three are not external adoption. ① UniPat evaluating others: the SaaS-Bench and RoadmapBench leaderboards include Claude Opus 4.7, GPT-5.4, Gemini 3.1 Pro, DeepSeek V4 Pro, GLM 5.1, Kimi K2.6, Qwen 3.6 Plus and MiniMax M2.7, and the BabyVision board includes Gemini3-Pro-Preview, GPT-5.2, Doubao-Seed-1.8 and Qwen3VL-235B. The direction is reversed. ② Reposting by third-party aggregators (BenchmarkList has leaderboard pages for RoadmapBench and BabyVision), which counts as half a step. ③ Mutual endorsement within the ecosystem (BabyVision = xbench × UniPat, and HongShan invested in both). ④ The real counter-evidence: the Qwen3-Coder-Next technical report (Mar 2026) uses SWE-Bench Verified / Multilingual / Pro + TerminalBench 2.0, and none of UniPat's benchmarks.
9. Cited is not trusted: three layers of credibility, and where to look
A benchmark's credibility is not one-dimensional. It can be split into three layers, and the US and China are at different levels of maturity in each:
| Layer | US | China |
|---|---|---|
| ① Cross-vendor common standards Who maintains them; are they in every lab's model card? | MMLU / GPQA / SWE-bench Verified: academic origins, adopted consistently by OpenAI, Anthropic, Google and xAI | C-Eval / CLUE / OpenCompass / CMMLU: originated at universities and national research institutes, formally cited in the DeepSeek-V3 and Qwen technical reports; firmly established |
| ② Vendor-built leaderboards Adopted by third parties beyond their own customers? | SEAL / APEX-Agents / GDP.pdf / Riemann-bench are cited in Anthropic and OpenAI system cards, but almost always as "customers citing their own supplier", with obvious selectivity | UniPat / xbench / Humanlaya: no evidence yet of citation in third-party model makers' technical reports; credibility shows up in funding markets rather than technical reports |
| ③ Public / media popularity boards Horizontal comparison for the public, not academic or commercial adoption | Chatbot Arena / LMArena: once the most credible, weakened in 2025–26 by disputes over "preference optimisation" vote gaming | SuperCLUE, Feixian Zhineng's ReLE: track mainstream model rankings continuously; credibility concentrated at the consumer-awareness level |
The two regions are structurally the same: what is truly used across vendors is always the non-commercial academic standard, and vendor leaderboards at most fill niches and serve as sales endorsements. The difference is maturity. US vendor leaderboards have made a small entry into top labs' official system cards, while the Chinese newcomers that started in 2025 still earn trust from capital markets.
When a third-party model maker that is not a customer cites a UniPat, Humanlaya or xbench leaderboard in its technical report, that will be the substantive signal that China's vendor-evaluation sector has moved from "fundraising narrative" to "industry adoption". It has not been observed yet.
Where to find more benchmarks
Agent-specific, most rigorous: HAL, the Holistic Agent Leaderboard (hal.cs.princeton.edu). Built by Princeton's SAgE team with an ICLR 2026 paper. It includes only 9 genuinely multi-step agentic benchmarks (no pure-knowledge tests like MMLU), and its core value is recording both the model and the scaffold, scoring each model × scaffold combination separately: 9 models × 9 benchmarks, 21,730 rollouts and about USD 40,000 in validation cost. It has stopped accepting new submissions and pivoted to "agent reliability" research (consistency, predictability, robustness, safety, refusal rate). Its academic credibility is solid (Princeton + ICLR, with API credits from OpenAI), but no leading lab has yet formally cited it in a system card.
| Live aggregator | Notes |
|---|---|
| Artificial Analysis | Compares models on intelligence, speed and price |
| OpenRouter Benchmarks | Based on real API traffic; leans toward agent and generation scenarios |
| llm-stats.com | Rankings of 300+ models |
| BenchLM.ai | Tracks 439 benchmarks in eight categories |
| Epoch AI Benchmarks | Collects third-party results and runs its own internal evaluations |
| AIMultiple | Claims 800+; widest coverage but uneven depth |
| Static curated list on GitHub | Notes |
|---|---|
| philschmid/ai-agent-benchmark-compendium | 50+ agent benchmarks; the clearest four-way classification |
| panilya/awesome-ai-benchmarks | 100+ benchmarks; broad coverage |
| supernalintelligence/Awesome-General-Agents-Benchmark | Focused on general agent capability |
| benchflow-ai/awesome-evals | A "non-BS" curated list |
10. Nine takeaways
- Benchmarks are the sales tentacles of an $8.5 billion training-data and RL-environment business, not standalone products. The grader (verifier) is the most expensive and central part of the chain, and that decides who has a motive to build benchmarks and who does not.
- Vendor leaderboards are sales funnels; academic leaderboards are public infrastructure. The first measures success by customers converted, the second by citations in papers and system cards. The US and China are structurally the same; China's gap is in maturity, not structure.
- Nearly half of benchmarks need a dedicated harness, mostly proprietary and unshared. Harbor is the rare exception, adopted by several vendors of different backgrounds.
- Harness and scaffold choice often moves scores more than the gap in model ability (up to 48 percentage points). It is now a research direction in its own right (Harness-Bench and others), not a fluke.
- Research benchmarks have gone from scarce to crowded but fragmented. At least 7–9 new research-agent benchmarks appeared in H1 2026 (from Meta, Microsoft, EPFL, the Shanghai AI Laboratory and others), but none has become a cross-lab standard written into system cards. The bottleneck has shifted from "nobody writes the questions" to "nobody agrees", and the openness and rubric subjectivity of research tasks make the harness sensitivity of §5 even more extreme than in coding (GAIA: the same 466 questions scored anywhere from 44.8% to 92.36% across leaderboards).
- Cited is not trusted; credibility must be read in three layers: common standards, vendor leaderboards and popularity boards. The US and China differ in maturity at each layer, so calling any leaderboard simply "authoritative" is inaccurate.
- China's new "evaluation-first" companies have not yet been adopted in third-party model cards. Their credibility comes from funding markets (Alibaba, Tencent, HongShan), not technical-report citations, and that is the key thing to watch. UniPat is no longer a pure evaluator (it trains UniScientist and UniMath), and "the question-setter is also a contestant, and the referee's taste has an origin" is a new kind of conflict of interest added in §8, harder to spot than the US pattern of customers citing their own supplier.
- The US and China have converged on the same method: rubrics. Open-ended tasks are broken into N independently judgeable items. The cost is quietly swapping the target from "right" to "complete": after rubric RL, presence items rise while factual correctness, concision, relevance and overall quality all fall, and rubric-free judges even prefer the pre-training base model. The gap is not grading precision (single-item judgement already matches human expert agreement) but the assumption that adding up N items equals quality. So for any rubric leaderboard, the first question is the grader's agreement with human experts, and only four of ten publish it.
- Only external ground truth can correct this substitution, and wet-lab experiments are not currently in any frontier model's RL loop. The three big labs use wet-lab work for post-hoc validation and data feedback (Anthropic goes furthest, with its own wet lab and funding for external validation). Only new companies that gave up generality, such as Periodic Labs and Lila, treat physical experiments as an RL environment, because the time scales differ by 3–5 orders of magnitude. Research benchmarks with ground truth are therefore necessarily small, slow and expensive, and their role should be calibrator, not leaderboard: a large, cheap rubric layer for daily iteration and a small, expensive ground-truth layer that periodically calibrates it. The calibration layer in between (regressing rubric weights on ground truth) has not been built by anyone.
Adapted from the Spark Science research notes (originally published in Chinese at jimmyresearch.com). Last updated Sep 20, 2026.