AI Infrastructure · Benchmark Supply Chain

Who Writes the Test, Who Grades It, and Can the Test Be Trusted?

A supply-chain map of 64 AI and agent benchmarks: vendor-built leaderboards vs academic standards, US vs China, the harness trap that moves scores by 48 points, and rubric grading.

Spark Science ResearchUpdated Sep 202655 min read

This is a study of the AI and agent benchmark supply chain: leaderboards built by data-service vendors versus public academic standards, the US compared with China, and a methodological trap most people have not noticed, namely the harness (or scaffold) used during evaluation.

"Model M scores 65 on leaderboard X" carries no information on its own, because you do not know what would happen with a different harness. Comparing two such numbers compares two systems, not two models.

64+
Benchmarks tracked in this piece
~47%
Need a dedicated harness to run
48 pt
Largest score gap for one model across harnesses
2
Harness ecosystems that do not interoperate

1. One benchmark, five independent dimensions

Before looking at any data, we set up an analytical framework. Every judgement below follows these five dimensions. They are independent of each other, and the combination of values a benchmark takes across them largely determines its business logic, its ceiling on credibility, and whether it can be used for objective comparison.

DimensionValuesWhat it determines
① Who sets the questionsCommercial data/environment company · academia or research institute · big-lab in-house (a lab sets questions to test itself and its peers) · platform/marketplace (sets no questions, only provides infrastructure)Whether a commercial motive exists, and whether there is a "referee who is also a player" conflict of interest
② What is evaluatedStatic knowledge/reasoning · agentic ability (multi-step, tool use) · vertical professional skill (investment banking, law, medicine, research)Whether the questions can be multiple choice, which indirectly sets cost and how far grading can be automated
③ How it is evaluatedPure API calls · requires a dedicated harness/environment · requires human reviewWhether the "change the scaffold and the score changes" trap applies (see §5)
④ OpennessFully open · partly public (to resist training contamination) · fully privateWhether third parties can reproduce results, and whether the question bank will be gamed
⑤ Regional ecosystemUS · China, each with its own mechanism for what counts as "accepted"Cited is not the same as trusted; the two regions build trust in entirely different ways (see §3 and §9)

Of the five, ① determines motive, ③ determines methodological risk and ⑤ determines the credibility ceiling. Those three are the threads this piece keeps returning to; ② and ④ are mostly descriptive labels.

2. Supply-chain economics: how a task is produced, priced and sold

To understand why so many commercial companies are rushing to build benchmarks, you first need the economics of the business itself. A benchmark is only the visible tip of the supply chain; underneath sits the whole industry of training data and RL environments.

$8.5B
Combined 2026 revenue of data/environment vendors
75%+
Revenue held by the "old four" (Scale, Surge, Mercor, Handshake)
$200–2,000
Typical production cost per task
4–5×
Premium for exclusive over non-exclusive rights

The four components of a sellable environment

  • An interactive world: a Docker container or micro-VM running real software or a replica of it.
  • Tasks: an initial state plus a prompt. These are not one-off runs but designed, reusable questions.
  • A grader (verifier): the most expensive and most central part. Three tiers, from most to least expensive: programmatic verification (unit tests, state checks) > rubric scoring > LLM judge.
  • Resettable and parallel: training needs dozens of rollouts per task, with thousands of containers running concurrently.

The cost goes into building the initial state and the grader. Experts must find real, difficult scenarios where frontier models genuinely fail, construct a clean reproducible environment, and write verification tests the model cannot cheat. This is also why benchmarks that need a harness (about 47% by the count in §4) are generally much more expensive and harder to build than pure knowledge tests.

Pricing structure

UnitPrice rangeNotes
Single taskUSD 200–2,000The mainstream range; complex software-engineering tasks occasionally reach USD 20,000
Website/UI gym replicaAbout USD 20,000 eachOpenAI reportedly buys browser environments at this price
High-fidelity product replicaUp to about USD 300,000 eachEnterprise software replicas of the kind Fleet builds
Quarterly contractSix to seven figures USD per quarterThe common form between labs and top vendors
Exclusivity premium4–5× the non-exclusive priceLabs do not want competitors to get the same data

Four production models

ModelExamplesCharacteristics
Handcrafted boutiqueMechanizeFull-time experts polish each task; depth first, "few but deep"
Assembly-line replicasFleet, Scale RL EnvironmentsSoftware replica templates plus tooling; one base spawns thousands of tasks and can be sold to several buyers
Platform + marketplaceHUD, Prime IntellectBuild no environments themselves; provide SDKs, hosting and matchmaking, "selling shovels to gold miners"
Expert-network question writingMercor, Handshake, SurgeTalent pools find experts to write questions, graded by humans or rubrics; moving toward automated environments

Domain distribution: coding is the holy grail, and science went from "empty" to "crowded but without a standard"

Coding is called the holy grail: long-horizon reasoning, tool use, statefulness, error recovery and dense feedback (compiling and testing act as the reward). It is where investment is most concentrated, followed by computer use and enterprise software. Medicine, law and finance rely mainly on expert-written questions.

Sep 2026 update

An earlier version of this section judged that "science is almost empty." That conclusion is out of date and needs correcting. In the first half of 2026 at least 7–9 new research-agent benchmarks appeared in quick succession (AutoResearchBench, ResearchClawBench, SciAgentArena from Microsoft, MLR-Bench, AIRS-Bench, HeurekaBench and others; see §6), from institutions as different as Meta, Microsoft Research, EPFL and the Shanghai AI Laboratory (InternScience). Questions are no longer scarce.

But the bottleneck has moved from "nobody writes the questions" to "nobody agrees." These benchmarks each cover a different slice of the research lifecycle (some test ideation and planning, some test execution, and they do not connect), and none has been adopted by several labs together or written into a system card. The root cause is no longer a lack of automated verifiers (rubrics plus LLM judges can already get around that). It is that research tasks are so open-ended and expert rubrics so subjective that the same question can swing widely in score depending on who grades it. This echoes the harness sensitivity in §5, and it is worse for research benchmarks (see the GAIA case in §6). The gap is still being filled not by annotation companies but by "the lab as reward function" (Periodic Labs) and "research software connected directly to the lab" (Anthropic × Benchling), which bypass the benchmark layer and use real experimental results as the training signal.

3. Who builds benchmarks, and why

AI benchmarks now have two entirely different kinds of producer. One is data and environment vendors. They sell training data and RL environments, and also set and grade their own tests; evaluation is essentially the first step of the sales funnel: diagnose where a model is weak, then sell you the data or environment to fix it. The other is academic and research institutions, whose goal is a shared public yardstick for the whole industry, with no commercial conversion attached. Being cited is their only "return".

Each kind has evolved its own system in the US and in China, giving four quadrants:

US · Vendor-built

  • Scale AI: SEAL Leaderboards (including SWE-bench Pro)
  • Surge AI: Hemingway-bench / AdvancedIF / GDP.pdf / Riemann-bench
  • Mercor: the APEX series (-1 / -SWE / -Agents / -Accounting)
  • Handshake AI: BankerToolBench / ATLAS-Finance / VIALS
  • HUD: OSWorld-Verified / SheetBench-50
  • Fleet and Mechanize have no separately branded public leaderboard
SEAL, APEX-Agents, GDP.pdf and Riemann-bench have been formally cited in Anthropic and OpenAI system cards.

US · Academic / community standards

  • MMLU / MMLU-Pro, GPQA Diamond, HLE
  • GSM8K, MATH, AIME
  • HumanEval, MBPP, SWE-bench Verified, LiveCodeBench
  • AgentBench, WebArena, τ-bench
  • Chatbot Arena/LMArena, AlpacaEval, MT-Bench
Reported by almost every frontier lab at every launch; the de facto common currency of the industry.

China · Vendor-built

  • UniPat AI: USD 300M Series A led by Alibaba in Sep 2026, valued at USD 2.5B
  • Humanlaya (formerly Rang Zhineng): One-Million Bench / BiomniBench
  • HongShan (formerly Sequoia China) xbench: ScienceQA / DeepSearch / AgentIF / BabyVision
  • Zhiyan Huisheng: Tencent has invested; little public information
  • Haitian Ruisheng (SpeechOcean) / Datatang / Testin Data: have evaluation services but no productised public leaderboard
Almost no evidence of formal citation in third-party model companies' technical reports.

China · Academic / national-team standards

  • C-Eval (Shanghai Jiao Tong + Tsinghua + Edinburgh)
  • CLUE / CLUEWSC, CMMLU
  • BAAI's FlagEval
  • Shanghai AI Laboratory's OpenCompass (Sinan)
  • SuperCLUE, Feixian Zhineng's ReLE
C-Eval, CLUEWSC, CMMLU and OpenCompass are formally cited in the DeepSeek-V3 and Qwen technical reports.

Laid on top of each other, the four quadrants show that the US and China are structurally the same: what is truly used across vendors is always the non-commercial academic standard, and vendor leaderboards at most fill niches and serve as sales endorsements. The difference is maturity. US vendor leaderboards have made it into some top labs' official system cards (even if only as customers showing off results), while the Chinese companies that started in 2025 still earn trust mainly from capital markets and have not yet been formally adopted in a leading model company's technical report.

Why vendors build their own benchmarks: an income statement

Not every vendor has one. Of the old four (Scale, Surge, Mercor, Handshake) plus the newer three, five (the old four and HUD) have built and branded public benchmarks. Fleet and Mechanize have not. They are precisely the two with the smallest teams and the most boutique, vertical strategy, putting all their limited expert capacity into the core product.

Benefits

  • Free authoritative endorsement: a citation in a lab's system card works like third-party certification.
  • Proof of production capability: being able to write questions that frontier models fail is itself a demonstration of expertise to buyers.
  • Defining the standard means owning the conversation: labs that want a score cannot avoid you.
  • Feeds the data flywheel: failure modes seen on the leaderboard directly shape the next batch of training tasks.
  • Differentiation: APEX focuses on professional tasks and BankerToolBench on investment banking, avoiding sameness.

Costs

  • The referee-player paradox: selling data while grading; recall that Scale's customers pulled orders en masse after Meta's investment raised doubts about its neutrality.
  • High maintenance: question banks get gamed and overfit, so questions must be rotated continually to prevent leakage.
  • Reputation is fragile: if questions are revealed to have leaked to one lab's training, the damage far exceeds the benefit.
  • Distraction from the core business: it is essentially marketing, with a risk of misallocated resources.
  • Backfire: it is awkward when your own customer's model ranks badly on your leaderboard.

A self-built benchmark is more of a brand-premium play for vendors that are already large and deep in experts than a standard feature of data and environment suppliers. That Fleet and Mechanize skip it confirms that a leaderboard is a nice-to-have, not a survival requirement.

4. The full catalog: 64 benchmarks, one by one

Each benchmark is recorded on the same six fields: size of question bank, openness, capability evaluated, host, citation by model companies, and evaluation method (pure API or dedicated harness). They are grouped by the four quadrants; click to expand.

US · Vendor-built16
BenchmarkSizeOpennessCapabilityHostCited byMethod
Scale SEAL Leaderboards~1,000 questions per sub-boardPrivateCode / Spanish / instruction following / mathScale AIAnthropic Opus 4.6Human review
Scale SWE-bench Pro1,865 (731 public + 858 held-out + 276 private)Partly publicEngineering fixes in real codebasesScale AINone foundNeeds harness
Surge Hemingway-benchNot disclosedPrivateCreative / business writing qualitySurge AINone foundHuman review
Surge AdvancedIF1,600+ promptsPrivateComplex multi-turn instruction followingSurge AI (with Meta MSL)No formal MSL citation foundExpert rubric
Surge GDP.pdf100 (10 professional fields)PrivateUnderstanding real professional documentsSurge AIOpenAI GPT-5.6; Anthropic Fable5/Mythos5Human review
Surge Riemann-bench25Methodology publicResearch-level frontier mathematicsSurge AIAnthropic Fable5/Mythos5API + automatic checking
Mercor APEX-1400 hidden + 100 open on HFPartly publicInvestment banking / consulting / law firms / doctorsMercorNone foundPure API + judge
Mercor APEX-SWE200 (50 open on HF)Partly publicReal software integration / debuggingMercorNone foundNeeds harness
Mercor APEX-Agents480Fully openLong-horizon cross-app professional-services agentMercor (Archipelago)Anthropic Opus 4.6Needs harness
Mercor APEX-Accounting160 tasks (10 "worlds", each with an accounting system + spreadsheets + PDFs)Tasks CC-BY; Archipelago environment open sourceAgentic accounting tasks (built with Ramp)MercorNone foundNeeds harness (Archipelago)
Handshake BankerToolBench100Fully openEnd-to-end investment-banking workflowsHandshake AI ResearchNone foundHarbor
Handshake ATLAS-Finance100 (13 financial simulation environments)Fully openMulti-app collaboration at financial institutionsHandshake AI ResearchNone foundHarbor
Handshake VIALS161Fully openLife-science visual understanding (VQA), part of the ATLAS suite familyHandshake AI ResearchNone found; released Aug 2026 (arXiv 2608.21357)API + LLM judge; best model still <30% accurate
HUD OSWorld-Verified369+ desktop tasksPublicOperating real desktop environmentsCreated by XLang Labs, maintained by HUDWidely tested by OpenAI, Google, Claude and othershud-python SDK
HUD SheetBench-5050Fully openSpreadsheet-analyst skills; CFO-level experts from PwC, Cisco, Schwab and Fannie Mae helped reviewHUD × Sepal AI (YC S24)None found; used internally by PwC and othershud-python SDK
Mechanize GBA EvalOne task (24-hour long horizon)Methodology publicExtreme long-horizon software engineeringMechanizeSelf-tested; no reverse citation foundDocker + goose

An interesting M&A crossover: Sepal AI (YC S24, a network of 20,000+ experts), which helped HUD build SheetBench-50, was acquired by Mercor on Feb 6, 2026. In other words, the team that built an evaluation set for one vendor was later bought by that vendor's competitor.

China · Vendor-built13
BenchmarkSizeOpennessCapabilityHostCited byMethod
xbench-ScienceQA100Fully publicScience Q&A / reasoningxbench.org (HongShan)None foundPure API
xbench-DeepSearch100Fully publicDeep search of the Chinese-language internetxbench.orgNone foundFixed ReAct agent (5 tools)
AgentIF-OneDay104 tasks / 767 scoring pointsFully publicAgent instruction following in everyday scenariosxbench.orgNone foundAny architecture; graded on deliverables
BabyVision388 + 20-question MiniCode public but no licence filePure visual understanding / reasoning (arXiv 2601.06521)xbench.org × UniPatJoint PR casePure multimodal API
UniPat Monthly-SWEBench~100 per month / ~20 repos, multilingualNo public repo (leaderboard page only)Real GitHub issues, refreshed monthly against contaminationunipat.aiNone foundHarbor + Terminus-2
UniPat SaaS-Bench23 systems / 106 tasks / 3,971 checkpointsOpen source, Apache-2.0GUI agents operating real SaaS (arXiv 2605.15777)unipat.aiNone foundOwn Docker + browser-use
UniPat DeepTerminalBench50 tasksOnly mentioned in the Terminal-X repo description; no separate repoSingle-turn deep engineering (the "depth" branch of Terminal-X)unipat.aiNone foundTerminal-Bench 2.0 format + Harbor
UniPat RoadmapBench115 tasks / 17 codebases / 5 languagesOpen source, MITCoding agents upgrading versions to spec (long horizon)unipat.aiNone foundHarbor + OpenHands
UniPat EvoCode-Bench26 tasks / 5–15 rounds / 227 evaluation roundsOpen source, MITMulti-round coding under changing requirements (arXiv 2605.24110)unipat.aiNone foundHarbor multi-turn fork
UniPat ClawBenchNot disclosedOpen source; ownership unclearBrowser agents on real websitesNAIL Group (UniPat a participant)None foundOpenClaw by default
Humanlaya One-Million Bench400 (200 Chinese + 200 English)Open sourceExpert tasks across 5 domains and 37 subdomainshumanlaya.comSelf-tested on 50+ modelsAPI + negative rubric
Humanlaya BiomniBench100 tasks (50 public + 50 private)Partly openProcess-level evaluation of biomedical research agentsxbench.org, with Phylo/HumanlayaNone found; companion to Stanford's BiomniHarbor + Terminus-2
Humanlaya ExcelBenchAfter further checking: Humanlaya's website, GitHub and Hugging Face show no separately released ExcelBench question bank or paper. It appears to be a media label for "Excel modelling tasks" rather than a formal product, so it is no longer counted as a separate benchmark.
Haitian Ruisheng internal evaluation systemNo public sizeNot open, B2BGeneral language + safety/reliability (6 capabilities, 19 sub-capabilities)Haitian Ruisheng (DOTS-LLM)None foundAPI + human review (inferred)
A name clash you must keep apart

There are two different SaaS(-)Bench papers. ① SaaS-Bench (arXiv 2605.15777) is UniPat's, Can Computer-Use Agents Leverage Real-World SaaS to Solve Professional Workflows?, with 23 systems and 106 tasks. ② SaaSBench (arXiv 2605.17526) is from USTC and Amap (Alibaba), Exploring the Boundaries of Coding Agents in Long-Horizon Enterprise SaaS Engineering, with 30 tasks and 5,370 verification nodes. They are not the same thing, and search results frequently mix them up. An added irony: Alibaba led UniPat's Series A, and an Alibaba team then published a benchmark with the same name. The "enterprise SaaS coding evaluation" niche is being claimed by several parties at once, and UniPat has no exclusivity there.

Openness in detail (measured via the GitHub API on Sep 20, 2026): of UniPat's 9 public repos, only 4 carry an explicit licence: SaaS-Bench (Apache-2.0), RoadmapBench (MIT), EvoCodeBench (MIT) and harbor_multiturn (Apache-2.0). BabyVision, UniScientist, SWE-Vision, Terminal-X and unimath have public code but no licence file (strictly speaking not open source, and legally risky for enterprise use). Monthly-SWEBench has no public repo. Star counts are telling too: the coding benchmarks have only 16–28, while the models and BabyVision have 172–253. Things you must clone to run get very little attention; things you can download directly or that make headlines get more.

US · Academic / community standards28
BenchmarkSizeOpennessCapabilityHostCited byMethod
MMLU14,042Fully publicKnowledge across 57 subjectsUC Berkeley et al.De facto universal standardPure API
MMLU-Pro12,032Fully publicHarder multi-task understandingTIGER-LabReplacement after MMLU saturatedPure API
GPQA Diamond198 (physics 82 / chemistry 78 / biology 38)Public (some access limits)PhD-level science problemsRein et al.Reported by OpenAI, Anthropic, Google and xAIPure API
BIG-Bench Hard23 tasks (>100 examples each)Fully publicSemantic / numerical / logical reasoningBIG-bench communityPartly replaced by GPQA/MMLU-ProPure API
HLE2,500 (100+ subjects)Fully publicGraduate-level frontier knowledgeCAIS + Scale AIAdopted by major vendors as the high-difficulty standardPure API
GSM8K8.5KFully publicGrade-school math word problemsOpenAISaturatedPure API
MATH12,500Fully publicHigh-school competition mathDan Hendrycks et al.Common standardPure API
AIME (as used for LLMs)~15 per yearFully publicTop competition mathMathematical Association of AmericaStandard for reasoning modelsPure API, needs multiple sampling
HumanEval164Fully publicPython function-level code generationOpenAIHighly saturatedCode sandbox (non-agentic)
MBPP974Fully publicIntroductory Python programmingGoogle ResearchOften paired with HumanEvalCode sandbox (non-agentic)
SWE-bench Verified500 (full set 2,000+)Fully publicFixing real GitHub issuesPrinceton + Stanford + UChicagoCommon standard for agentic codingDocker; custom scaffolds allowed
LiveCodeBenchRolling ~400–600Fully publicCompetitive programming, rolling updates against contaminationUC Berkeley groupOften cited by vendors stressing decontaminationCode sandbox (non-agentic)
AgentBench8 environmentsFully publicAgent reasoning and decisions across environmentsTHUDM (Tsinghua)Widely cited in academiaUnified text protocol
WebArena6 self-hosted sites / 812 tasksFully publicReal web navigation and operationCMURecognised academic standardSelf-deployed sites; scaffolds not unified
τ-bench (incl. Airline)Dozens to 100+ tasksFully publicTool use + multi-turn customer-service interactionSierraListed on OpenRouter (123 models)Officially separates standard and custom submissions
RULERFour kinds of synthetic taskFully publicReal effective context lengthNVIDIACommon long-context referencePure API
NIAHA method, no fixed bankFully publicLong-text retrievalCommunity (Greg Kamradt)Saturated, replaced by RULERPure API
LongBench21 datasets (Chinese and English)Fully publicLong-document QA, summarisation and 6 categories in allTsinghua KEG (Zhipu's predecessor)Commonly cited for Chinese long contextPure API
MMMU11.5K (30 subjects)Fully publicMultimodal university-level understandingAcademic teamMultimodal standardPure API (image + text)
MathVista6,141Fully publicMath reasoning in visual contextsAI4Math academic teamCommonly cited for multimodal mathPure API (image + text)
Chatbot Arena / LMArenaNo fixed bank; crowdsourced votesPlatform open sourceOverall conversational preferenceStarted at LMSYS; spun out as a for-profit in 2026Once the most credible; declined amid vote-gaming disputesHuman voting platform
AlpacaEval805 fixed instructionsFully publicSingle-turn instruction-following preferenceStanfordCommonly cited in academiaAPI + LLM judge
MT-Bench80 multi-turn question setsFully publicMulti-turn conversationLMSYSValidation set for the LLM-as-judge methodAPI + judge
TruthfulQA817Fully publicTruthfulness against common misconceptionsAcademic teamCommon safety/truthfulness comparisonPure API
SimpleQA4,326Fully publicAccuracy on short factual questionsOpenAIStandard for factuality / hallucination ratePure API
BrowseComp1,266Fully publicMulti-step web browsing to locate informationOpenAICommon for Deep Research-style productsNeeds search tools; scaffolds not unified
DeepSearchQA900 (17 domains)Open (HF: google/deepsearchqa, Apache 2.0)Multi-hop deep-research QA with "causal chain" questionsGoogle DeepMind (not an independent academic team)Listed on OpenRouter; few models testedNeeds search tool calls
WideSearch200 (100 Chinese + 100 English)Open (GitHub + HF)Breadth-first information gatheringByteDance Seed team (not Western academia)Few models testedNeeds search/browse tools

Correction: on further checking, DeepSearchQA and WideSearch are not products of independent academic teams but in-house evaluations by Google DeepMind and ByteDance Seed respectively. Strictly, they belong in a fifth category this piece does not break out, "big-lab in-house", which is neither pure academic community nor a third-party vendor as defined here.

China · Academic / national-team standards7
BenchmarkSizeOpennessCapabilityHostCited byMethod
C-Eval13,948 (52 subjects)Fully openKnowledge and reasoningShanghai Jiao Tong + Tsinghua + EdinburghDeepSeek-V3, Qwen2.5Pure API
CLUE / CLUEWSC9 CLUE tasksFully openChinese NLUCLUE academic communityDeepSeek-V3, Qwen2.5Pure API
BAAI FlagEval90+ datasets / 2M+ itemsOpen platform, partly open sourceLanguage / multimodal image-text / text-to-image and text-to-videoBeijing Academy of AI (BAAI)No formal citation found; BAAI publishes its own rankingsSome need environments
OpenCompass (Sinan)70+ datasets / 400K+ itemsFully openLanguage / reasoning / knowledge / code / math / agentsShanghai AI LaboratoryExplicitly cited in Qwen technical reportsPartly (agent dimension)
SuperCLUEThree benchmarks, hundreds to thousands of itemsOpenGeneral + specialised + industry skillsSuperCLUE team (independent of CLUE)Not cited in its own subjects' reports; a third-party tracking boardNeeds interactive environment
ReLE (Feixian Zhineng)374 models / ~300 fine-grained dimensionsOpenEducation / medicine / finance / law / agentsFeixian Zhineng (independent community)None found; aimed at media and communityPartly (agent dimension)
CMMLU11,528 (67 subjects)Fully openKnowledge and reasoning in Chinese contextsIndependent academic team (published at ACL)Explicitly cited by the Qwen seriesPure API

Counting rules: 30 need a dedicated harness or environment (~47%), 3 partly need one, 24 are pure API, 4 need human review, 1 is a special platform (Chatbot Arena) and 2 have insufficient information. HumanEval, MBPP and LiveCodeBench do need an execution environment, but they are single-turn code generation plus sandbox tests with no multi-step agent scaffold, so the harness-sensitivity problem in the next section does not apply to them.

5. The harness trap: same model, different scaffold, up to 48 points apart

Harness and scaffold are not the same, but the industry uses them interchangeably

According to Hugging Face's agent glossary, the scaffold is the behaviour-definition layer: the system prompt, tool descriptions, how responses are parsed and what is remembered across steps, all settled before the first prompt. The harness is the execution layer: calling the model, dispatching tool calls and deciding when to stop, everything after the first prompt. The popular formula is Agent = Model + Harness. In practice the two are often treated as synonyms; Claude Code's official documentation calls itself "the agentic harness around Claude", folding scaffold duties into the harness. Like most papers, this piece calls both layers "harness".

Three layers: environment / agent (scaffold) / model

A benchmark run actually stacks three layers: the environment infrastructure (sandbox, Docker, deployable systems, and who maintains them) → the agent or scaffold (a standardised default, or bring-your-own, such as Terminus-2 vs Claude Code) → the model (the LLM actually being evaluated). The environment infrastructure we found splits into two ecosystems that do not interoperate:

The Harbor ecosystem

Open-sourced by the Laude Institute (creator of Terminal-Bench) under Apache 2.0. A unified task.toml format explicitly decouples the environment, agent and model layers. Built-in agents include claude-code, codex, aider, gemini-cli, opencode, OpenHands, goose and mini-swe-agent.

  • BankerToolBench, ATLAS-Finance (Handshake)
  • BiomniBench (Humanlaya; fixed harness = Terminus-2)
  • Monthly-SWEBench, RoadmapBench, EvoCode-Bench (UniPat's Terminal-X suite)
  • Terminal-Bench, SWE-Bench adapter layer
Currently the only infrastructure where the agent layer is genuinely shared across different vendors.

Each vendor's own stack

These also separate environment from agent, but design their own agent abstraction and do not share it.

  • Scale: three-layer Docker images for SWE-bench Pro
  • Mercor: Archipelago (in-house, for APEX-Agents; default ReAct toolbelt)
  • HUD: hud-python SDK (OSWorld-Verified / SheetBench-50)
  • Mechanize: own Docker (GBA Eval; one known submission used Claude Sonnet 5 + goose)
Building and maintaining a harness is a core asset and moat for vendors; standardising it would erode their differentiation.

Hard data: score swings far exceed gaps in model ability

CaseFixed / standard harnessNative / custom harnessGap
Claude Opus 4.7 · BiomniBenchTerminus-2: 63.94Claude Code: 73.349.4 pts
Claude Opus 4.5 · SWE-bench ProSEAL scaffold: 45.9%Claude Code: 55.4%~10 pts
Similar models · SWE-bench Verifiedmini-SWE-AgentOpenHands: 77.6%Several points
A model on WebArenaOriginal harness: 25%Enhanced agentic scaffold: 55%30 pts
HAL statistics · SWE-bench Verified MiniTop models across different scaffoldsUp to nearly 48 pts
0 25 50 75 100 BiomniBench +9.4 SWE-bench Pro +9.5 WebArena +30 GAIA (466 qs) +47.6
Score ranges (0–100) for the same model or the same question set under different harnesses or leaderboards. Grey dot: standard/fixed harness or lowest reported score; coloured dot: native/custom harness or highest. GAIA is discussed in §6.

The heart of the problem: "A leaderboard entry that reads 'model M, SWE-Bench Verified 65%' carries no information on its own, because you do not know what would happen with a different scaffold. Comparing two such numbers compares two systems, not two models." (From the paper Stop Comparing LLM Agents Without Disclosing the Harness, arXiv 2605.23950.)

Can vendors choose their own harness? Yes, and it is the default

  • SWE-bench officially allows custom scaffolds (AutoCodeRover, Agentless, SWE-Agent and OpenHands all appear on the leaderboard). Each entry is really a "model + harness" system competing.
  • Anthropic often reports scores with Claude Code directly. On Terminal-Bench 2.0, OpenAI is the only participant that insists on its own Codex CLI rather than the official Terminus-2.
  • In Nov 2025 SWE-bench tightened its policy: Verified and Multilingual accept submissions only from academic institutions, and independent or commercial submissions are limited to the lower-bar Lite board. This is probably a response to commercial companies using custom harnesses to inflate scores, but Lite still allows custom harnesses.
  • τ-bench is the only benchmark that institutionalises "standard vs custom submissions": replacing the default scaffold (adding a planner, extra agents, custom tools or a changed control flow) must be labelled separately.

Common open-source harnesses

HarnessCategoryMaintainerNotes
SWE-agentCode / SWEPrinceton-affiliated academic teamIntroduced the agent-computer-interface paradigm; most cited
OpenHandsCode / SWE; general agent platformOpen-source communityShips its own SWE-bench adapter; feature-heavy
mini-SWE-agentCode / SWEOpen-source communityMinimal harness, often used as the standardised control
Terminus-2Terminal / CLI tasksTerminal-Bench (Harbor)Default standard harness for Terminal-Bench 2.0
Agent-S / S3Desktop / computer useSimular (open source)First agent to beat humans on OSWorld (72.6%)
UI-TARSDesktop / GUI groundingByteDance (open source)Trained end to end; treats UI elements as first-class citizens
τ-bench harnessCustomer service / tool useSierra (open source)Simulated user + tools + database interaction
HarborGeneral evaluation / RL environment frameworkLaude Institute (open source, Apache 2.0)Standardised infrastructure shared by several vendors
gooseGeneral coding agentBlock (open source)Used as the native harness for Mechanize's GBA Eval and others
OpenCodeGeneral coding agentOpen-source community (MIT, 208k+ stars)One of Harbor's built-in agents; supports 75+ model providers

Harnesses the big model companies use for their own tests

CompanyUsual harnessNotes
AnthropicClaude Code (own product)About 10 points above the standard scaffold on SWE-bench Pro
OpenAICodex CLI (own product)The only Terminal-Bench 2.0 participant not using the official Terminus-2
GoogleGemini CLI + an open-source computer-use-preview reference implementationMainly for browser/GUI; desktop operation officially "not yet optimised"
Leading Chinese models (DeepSeek / Qwen / GLM / Kimi)No public disclosure foundAgentic evaluations rarely state the scaffold; knowledge tests use the bare API
A counter-intuitive finding

The Holistic Agent Leaderboard (HAL) ran 21 controlled model-harness comparisons (Claude Code / Codex CLI / Pi, covering 7 models on SWE-bench Lite and Terminal-Bench 2.0). In 9 of 12 comparisons, switching to a harness that was not the vendor's own produced a higher score. A vendor's own harness is not necessarily the objectively strongest; it is more likely just the one it has tuned most carefully for itself.

6. AI for Science: no shortage of questions, but no shared standard

Nine in ten of the benchmarks on our list that carry a "science" label (GPQA, HLE, Riemann-bench, xbench-ScienceQA, MMMU, MathVista) measure scientific knowledge: whether a model understands science, not whether it can do research. Benchmarks that measure the process of AI doing research used to be extremely scarce, but that judgement was overturned in the first half of 2026 as a wave of new benchmarks appeared. The table below gives the overview; we then go deep on the heaviest of them, BiomniBench.

New research-agent benchmarks from H1 2026

BenchmarkReleasedHostSizeWhat it testsOpen
AutoResearchBenchApr 2026 (arXiv 2604.25256)Academic team–Scientific literature discovery (Deep Research single-paper tracing + Wide Research batch retrieval); not a process evaluationOpen
ResearchClawBenchJun 2026 (arXiv 2606.07591)InternScience (Shanghai AI Laboratory / OpenGVLab)40 tasks / 10 scientific fieldsBased on real published papers with the target paper hidden; expert multimodal rubric scoring. Same approach as BiomniBench but broader coverageOpen
SciAgentArenaJun 2026 (arXiv 2606.12736)Microsoft Research~200 tasksCross-scale research (single-cell omics, spatial omics, drug discovery, EHRs, genetics) with stepwise verification; 18 agents testedUnclear
MLR-BenchMay 2025 (arXiv 2505.19955), accepted at NeurIPS 2025Academic team201 tasks (from NeurIPS/ICLR/ICML workshops)The full idea → proposal → experiment → paper pipeline, auto-reviewed by MLR-JudgeOpen
AIRS-BenchFeb 2026 (arXiv 2602.06855)Meta / Facebook Research20 tasks (from SOTA ML papers)The whole research lifecycle (ideation → experimental analysis → iteration), with no baseline code providedOpen
HeurekaBenchICLR 2026EPFL (mlbio-epfl)Instantiated as sc-HeurekaBench (single-cell biology)Really a framework for generating benchmarks: extract insights from papers → turn into QA → agent analyses and verifies autonomouslyOpen
StatefulDiscoveryJun 2026 (arXiv 2606.11851)Academic teamReuses 40 real-data tasksStrictly an evaluation of a method / agent architecture (explicit discovery-state control), not a standalone question bank–

Similar projects also found: InquiTree (arXiv 2606.09550, evaluating paper-derived "research tree" scientific inquiry), AstaBench (arXiv 2510.21652) and SoundnessBench (arXiv 2605.30329, testing whether AI can tell good research ideas from bad).

China's science evaluation landscape

China's science benchmarks cluster around the Shanghai AI Laboratory, plus projects involving HongShan's xbench and Humanlaya. They are characterised by systematic platforms more than single process-level evaluations:

BenchmarkHostWhat it testsLevelAI performance
Sinan Scientific Intelligence EvaluationShanghai AI Laboratory (OpenCompass)Evaluation platform covering the whole research workflowPlatformConclusion: clear weaknesses in models' scientific ability
SFE (Scientific Foundation Evaluation)Shanghai AI Laboratory66 tasks, 5 scientific fields, bilingual; three levels: perception → attribute understanding → comparative reasoningKnowledge / reasoning (multimodal)Mainstream models score only ~30
SciHorizonAcademic team (arXiv 2503.13503)Tests 50+ LLMs on 16 dimensions across math, physics, chemistry, life sciences and earth/space; also assesses whether scientific data is "AI-ready"Knowledge / reasoning + data readinessSystematic diagnosis, no single headline score
SciEvalAcademic team (NSFC-funded)Multi-level science Q&AKnowledge / reasoning–
ResearchClawBenchInternScience (Shanghai AI Laboratory)40 tasks / 10 fields, hidden target paper, expert rubricResearch processBest system (Claude Code) only 21.5/100
xbench-ScienceQAHongShan xbench100 science questionsKnowledge / reasoningNo third-party citations seen
BiomniBenchStanford + Phylo + Humanlaya (a Chinese data lab took part)See belowResearch processFrontier models 62–73

How well does AI do in science? Four judgements

  1. "Knowing science" and "doing research" are far apart. Multiple-choice GPQA is close to saturated, but once an agent must choose methods and interpret results itself, scores drop into the 20–70 range (ResearchClawBench 21.5, BiomniBench 62–73).
  2. The biggest problems are fabrication and shallowness. MLR-Bench found that 80% of cases fabricated or could not verify results; BiomniBench commonly sees key analysis steps skipped; SciAgentArena found agents could not produce novel insight. Work that looks complete but does not hold up scientifically is more dangerous than a wrong answer.
  3. AI has roughly caught up with junior scientists but not senior ones. BiomniBench's early human baseline: senior pharma scientist 68.5, best AI 65.0, junior scientist 48.5.
  4. The shared bottleneck in the US and China is the lack of a consensus standard. The US is strong in heavily resourced process-level evaluations from several institutions (Meta, Microsoft, Stanford, FutureHouse); China is strong in systematic platforms (Sinan, SFE, SciHorizon). Neither side has a research standard that several leading labs have written into their system cards.
Scores are low, and none has become a cross-lab standard

AutoResearchBench's Deep Research accuracy is only 9.39%. The best system on ResearchClawBench (Claude Code) averages only 21.5/100. MLR-Bench found coding agents fabricated or could not verify experimental results in 80% of cases. On AIRS-Bench, agents beat the human SOTA on only 4 of 20 tasks. More importantly, we found no case of any of these benchmarks being formally cited in a major model company's system card. Academic analysis states plainly that "research-agent benchmarks remain fragmented, covering different slices of the research lifecycle (ideation/planning vs execution) that do not connect."

A harness-sensitivity case more extreme than BiomniBench: GAIA

GAIA is a general agent benchmark (not research-specific), but in May 2026 it produced a startling data point: on the same 466 questions, scores on different leaderboards ranged from 44.8% to 92.36%. That gap is not measurement error; it comes down to which tools or scaffold were allowed. It is even more extreme than the 10–48-point harness gaps for SWE-bench and BiomniBench in §5, and the logic is the same: research-type tasks are more open-ended and their rubrics more subjective, so harness sensitivity will only be worse than for coding benchmarks, not milder.

BiomniBench: still the heaviest example (no longer the only one)

The new benchmarks above show that research-agent evaluation has gone from empty to crowded, but BiomniBench remains the most heavily resourced of them: 100+ senior experts, over 1,000 hours of development, a direct companion to Stanford's production research agent Biomni, and a six-dimension rubric plus a two-track design (fixed harness vs native harness) that is the most complete among these new projects. The details:

Basics
  • Scale: 100 data-analysis tasks (50 public + 50 private against contamination), 17 task types, 5 disease areas plus general biology.
  • Source of questions: each is based on a real paper published in Nature, Cell or Science, developed with the original authors or domain experts.
  • Effort: 100+ senior experts and over 1,000 hours from Stanford, Harvard, Peking University and several leading pharma companies, together with Phylo and Humanlaya Data Lab.
  • Companion to: Biomni, the general biomedical AI agent from Stanford's SNAP lab (105 software packages + 59 databases + 150 specialised tools).
  • Core innovation: process-level evaluation. The agent's full trajectory is scored against an expert rubric, not just the final answer.

Scoring dimensions and examples

DimensionCore checkExample criterion
Data handlingLoads and prepares data correctlyLoads the required files, merges on the right identifiers, filters to the right treatment group
Method choiceChooses an appropriate analysis methodFor survival analysis, uses Kaplan-Meier curves + log-rank test
Statistical rigourApplies statistical tests correctlyApplies multiple-testing correction (BH-FDR)
Source reliabilityRelies on trustworthy referencesProvides relevant citations from peer-reviewed literature
Scientific reasoningLogical consistency and biological connectionAvoids overclaiming causation; links findings to real mechanisms
Biological interpretationSixth dimension added in the final version–

Typical failure modes: skipping the MYC gene-specific analysis, skipping the phenotype-correlation analysis, skipping the key pathway interpretation. The tasks are not "run a script and output a number"; the agent has to know which direction to dig.

Which harness is used

Underneath, the Harbor execution framework standardises the environment (task.toml configures verifiers and environment budgets), so data, tools and sandbox resources are identical across agents. Grading is not an automated verifier but an LLM judge. The paper compared how well 5 candidate judges (Claude Opus 4.6, DeepSeek V3.2, Gemini 3.1 Pro, GPT-5.4, Qwen 3.6) agreed with internal experts A/B/C and chose Gemini 3.1 Pro as the judge throughout (closest to the human experts).

How the models scored

Official fixed-harness comparison: with every model on Terminus-2, the top models were within 1.6 points of each other, clustered at 62–64. Official native-harness comparison (each vendor's own tuned agent):

CombinationScore
Claude Code + Opus 4.773.34 (highest)
Claude Code + Opus 4.669.51
Codex CLI + GPT-5.468.69
Codex CLI + GPT-5.567.59
Claude Code + Sonnet 4.666.39
Terminus-2 + Opus 4.7 (fixed-harness control)63.94

Early-version human baseline (v0, 15-task subset, published on Phylo's blog):

RankSystem or personScore
1Senior pharma scientist (5+ years)68.5
2Biomni Lab65.0
3Edison Analysis52.7
4OpenAI Agents SDK (GPT-5.2)51.4
5Junior scientist (3 years)48.5
6Claude Code (Opus 4.6)47.8

At the time, the best AI system (65) was close to or above the junior scientist (48.5) but clearly below the senior scientist (68.5). Direct "AI vs real expert" comparisons like this are rare.

Independent third-party replication (the omicverse team ran its own agent, "omicos", with different base models on the 50 public tasks): GPT-5.5 78.8%, DeepSeek4-pro 73.0%, GPT-5.4 67.7%, DeepSeek4-flash 66.0%, MiMo-v2.5-pro 65.8%, MiMo-v2.5 62.3%, GPT-5.4-mini 44.2%. A counter-intuitive finding: GPT-5.5 had the highest total but only 60% on biological interpretation, while some cheaper models (DeepSeek4-flash 76%, MiMo-v2.5-pro 73%) beat it on that dimension. Models generally cite their sources honestly, but consistently fall short on method choice and biological interpretation, the two dimensions that need real scientific judgement.

Not benchmarks, but worth watching

FutureHouse's LAB-Bench (2,457 questions, 8 categories and 30 subtasks, testing basic biology research skills including protocol troubleshooting in ProtocolQA and molecular-cloning scenarios; Anthropic reports scores on it) is another process/task-level evaluation outside the table above. Periodic Labs' "lab as reward function" (the model reads literature → runs quantum-chemistry calculations → an automated lab synthesises → measured results feed back as training signal) is a training paradigm rather than a benchmark, and sidesteps evaluation fragmentation altogether; for the full breakdown, and why the three big labs cannot do it, see §7. Anthropic × Benchling (research software connected directly) likewise bypasses the third-party benchmark layer.

7. Rubric grading: turning "unverifiable" into "verifiable", at the cost of swapping "right" for "complete"

The last section concluded that research benchmarks are crowded but lack consensus. Look only at method, though, and the US and China have converged on the same solution: break an open-ended task into N small, independently judgeable rubric items, then aggregate them into a score or reward. This section covers three things: how the mechanism works, how far it is from a true standard, and which methods have been shown to narrow the gap.

Rubrics make open-ended tasks gradable, but they reward "mentioned it", not "got it right", and they do not penalise "said too much". In medical Q&A that substitution costs little; in research it can be fatal. A research report that mentions everything but reaches the wrong conclusion will get a high rubric score.

Not new: a timeline of the paradigm

The standard names are rubric-based reward or rubrics as rewards. The motivation is that RLVR (reinforcement learning from verifiable rewards) has been hugely successful in math and code but cannot be used for tasks without a single answer, such as medical or scientific writing. The underlying assumption, that "verifying is cognitively much cheaper than creating from scratch", comes from the older scalable-oversight line of work (CriticGPT, debate-based alignment) and is not original to any new company.

DateWorkKey contribution
2024OpenAI Rule-Based Rewards / CriticGPTDeriving rewards from rule text; model-assisted human verification
May 2025OpenAI HealthBench262 physicians (practising in 60 countries), 5,000 conversations, 48,562 physician-written rubric items, weighted, graded by GPT-4.1
Jul 2025Scale AI Rubrics as Rewards (RaR), arXiv 2507.17746First systematic framework: four weighted item categories + GRPO; released RaR-Medical-20k / RaR-Science-20k
Aug 2025Xiaohongshu RL with Rubric Anchors, arXiv 2508.12790The largest rubric reward system at the time, 10,000+ items, explicitly sourced through human-machine collaboration; open-sourced Qwen-30B-A3B
Sep 2025Chasing the Tail, arXiv 2509.21500Theory locating the root of reward over-optimisation in high-score tail mismatch
2026Open Rubric System / Autorubric / OpenRubrics / ARBOR / SibylSense / RUC-NLPIR Rubrics SurveySigns of a mature paradigm: surveys appear, along with papers dedicated to failure modes

A direct implication for §8: UniPat's blog claims that "the two existing paradigms (purely human / purely synthetic) do not exploit this asymmetry." That is inaccurate against the literature. Scale's RaR came more than six months earlier, and Xiaohongshu's Rubric Anchors seven months earlier, explicitly describing human-machine collaboration. UniPat's real differentiation lies in domain coverage (50+ disciplines) and expert time per sample (1–2 hours), not in method.

What a rubric item actually looks like (examples from the RaR paper appendix)

Each question gets 7–20 rubric items, each with a title (2–4 words), a description (one sentence, starting with its category) and a weight. Items must be self-contained enough that "a non-expert can judge from this item alone, without inference or external sources". That is the precondition for an LLM judge to act as grader.

CategoryWeightMeaningExample from the paper (medical question)
Essential5Key fact or safety check; if missing, the whole answer is invalidEssential Criteria: Identifies non-contrast helical CT scan as the most sensitive modality for ureteric stones.
Important3–4Key reasoning, completeness, clarityImportant Criteria: Explains that non-contrast helical CT detects stones of varying sizes and compositions.
Optional1–2Style or extra depthOptional Criteria: States "The final answer is (B)" or similar answer choice formatting.
Pitfall−1 / −2Common errors or omissions for this question; negative pointsPitfall Criteria: Does not mention identifying (B) as the correct answer.

RaR's experiments on Qwen2.5-7B produced two counter-intuitive results. Implicit aggregation (handing the whole rubric to the judge for one overall score) beat explicit aggregation (judging item by item and summing weights), contrary to the intuition behind atomic verifiability. And in the science domain (GPQA-Diamond), RaR-Explicit did worse than the Reference-Likert baseline (0.3030 vs 0.3775). The benefit of rubrics depends on whether the task is truly open-ended; for multiple-choice tasks, decomposition adds little.

Five mechanisms research benchmarks use for rubrics

A · Weighted rubric tree (most fine-grained)

PaperBench (OpenAI, arXiv 2504.01848): replicating 20 ICML 2024 papers. The rubric is a requirements tree whose leaves are single pass/fail conditions, with each node hand-weighted relative to its siblings, 8,316 leaf nodes in all. The decomposition rule is strict: keep splitting until an expert can judge a node within 15 minutes, turning atomicity into an operational cost definition. Rubrics were co-written with each paper's original authors; grading uses o3-mini as judge, and a separate JudgeEval was built specifically to evaluate the grader.

B · Fixed-point checklist + pass mark (simplest)

FrontierScience-Research (OpenAI, arXiv 2601.21165): 60 original research subtasks designed by PhD scientists (doctoral students, professors, postdocs). Each has a 10-point rubric of independent, objectively judgeable items, each with a pass/fail description and points (for example "writes out equation X"); ≥7/10 counts as success. It drops exact matching in favour of evaluating intermediate reasoning steps. It is also the leaderboard UniPat's UniScientist reports on.

C · Dimension system + dynamic weights (for open-ended reports)

DeepResearch Bench (arXiv 2506.11763, ICLR 2026) uses the RACE framework: four orthogonal dimensions (comprehensiveness, insight depth, instruction following, readability); the judge generates task-specific weights per question (averaged over T trials) and then scores relative to a high-quality reference report. DEER (arXiv 2512.17776, ICML 2026): an expert taxonomy of 7 dimensions / 25 sub-dimensions → 101 fine-grained rubric items, plus a claim-verification architecture that checks both cited and uncited assertions. BiomniBench's six-dimension process-level evaluation belongs here too (see §6).

D · Atomic fact decomposition (no pre-written rubric)

HeurekaBench (ICLR 2026, arXiv 2601.01678): both the agent's answer and the ground truth are decomposed into atomic facts and compared for presence and correctness. GPT-4o grading this way reaches Spearman ≈ 0.90 against PhD-level experts. It is currently the only design that closes the "only checks whether it was mentioned" loophole at the mechanism level, because it also requires correctness.

E · Rubric + hard gates + real experiments (the only link to the physical world)

BioLab Bench: task-specific metrics + rubric verifiers + equipment-level validity gates + real experimental tests, chained in four layers. A cheaper alternative is BenchBench-Protocol (arXiv 2608.23898): 149 tasks drawn from 96 published protocols on Benchling, where the difference between the published version and the scientist-modified version is the ground truth, using completed wet-lab experiments as delayed ground truth.

The anti-LLM camp · an unresolved split

ResearchRubrics (Scale × UChicago, arXiv 2511.07685): 101 prompts / 2,593 human-written items, explicitly forbidding LLMs from generating or seeding any item, on the grounds that "having humans only verify LLM-generated rubrics introduces anchoring bias". Triple review and 2,800+ human hours. This directly conflicts with the "LLM generates, humans verify" route of UniPat and HeurekaBench, and nobody has run a head-to-head experiment on the same questions.

BenchmarkScaleWho writes the rubricMechanismGrader reliability published?
PaperBench20 papers / 8,316 leavesCo-written with original authorsWeighted requirements tree, 15-minute ruleYes, JudgeEval evaluates the grader
FrontierScience-Research60 subtasksPhD scientistsIndependent 10-point items, ≥7 to passNone found
ResearchRubrics101 prompts / 2,593 itemsHuman only; LLM generation bannedMulti-axis + triple review, 2,800+ human hoursRelies on triple review
DeepResearch Bench100 tasks / 22 fieldsGenerated dynamically by the judgeRACE: 4 dimensions + dynamic weights + relative scoringPAR 71.33% vs human inter-rater 68.44%
DEER50 tasks / 101 itemsExpert taxonomy7 dimensions, 25 sub-dimensions + claim verificationTask-specific expert guidance
BiomniBench-DA100 tasksOriginal authors or domain expertsSix-dimension process level, whole trajectoryCompared 5 candidate judges for agreement (see §6)
HeurekaBenchOEQ + MCQLLM-generated, verified against published conclusionsAtomic facts: presence + correctnessSpearman ≈ 0.90
SciAgentArena~200 tasksNot examinedStepwise verificationNone found
BioLab BenchNot disclosedWet-lab expertsRubric + equipment gates + real experimentsGround truth from experiments
HealthBench (medical; origin of the method)5,000 conversations / 48,562 items262 physiciansWeighted items, graded by GPT-4.1Agreement with physicians ≈ physician inter-rater agreement

When you look at any rubric leaderboard, the first question is not "who is on top" but "how well does your grader agree with human experts?" Only four of the ten in the table give a number.

How far from a true standard: two gaps, and the second is structural

The breakdown comes from Reward Hacking in Rubric-Based RL (arXiv 2605.12474): the policy optimises against a training grader but is evaluated by a panel of three frontier models from different families.

Gap one: grader failure (controllable). The training grader says an item is satisfied; the reference graders say it is not. Weak graders produce large spurious gains that do not transfer, and exploitation grows as training proceeds, concentrated in three recurring failures: partially satisfied compound conditions counted as passes, implied content treated as explicit statements, and loose topic matching. A stronger grader reduces this sharply but cannot eliminate it.

Gap two: the limits of rubric design itself (structural). Even with a very strong rubric grader, the responses it prefers look worse to a judge who ignores the rubric and assesses overall quality:

Change after rubric RLDirection
Completeness and "mentioned X" presence itemsUp
Factual correctnessDown
ConcisionDown
RelevanceDown
Overall qualityDown
The most striking finding

Rubric graders prefer the post-RL checkpoint, while rubric-free judges prefer the pre-training base model. The rubric score went up and the model got worse. The mechanism is easy to see: a rubric is a checklist, so the model's optimal strategy is to mention every item, producing output that is long, exhaustive and verbose, with lower factual density. The paper concludes that stronger verification reduces reward hacking but does not by itself guarantee that rubric gains correspond to broader quality improvements.

The counterpoint: single-item judgement is already accurate

HealthBench's meta-evaluation (physicians re-reviewing 34 consensus items, criterion-level Macro-F1) shows that pairwise agreement between the model grader and physicians matches agreement among physicians themselves. Put the two results together for the full picture: rubric graders already judge single items at human level; the problem is the aggregation assumption that adding up N items equals quality. The gap is not precision but metric structure.

Methods shown to narrow the gap (ordered by strength of evidence)

  1. Cross-family grader panels. Strictly separate the training grader from the evaluation grader; the hard rule is never let the generating model and the grading model come from the same family. Autorubric and PoLL-style ensembles show that when variance and threshold reliability matter, a jury of diverse judges beats a single strong model.
  2. Human-label calibration plus periodic recalibration (the most mature in engineering). Hand-label 100–300 representative outputs with the same rubric the grader will use, compute Cohen's kappa, Pearson and MAE, and target kappa > 0.6. Recalibrate quarterly to catch drift, and always recalibrate after the grading model is upgraded.
  3. Pairwise comparison instead of pointwise scoring. Open Rubric System (arXiv 2602.14069) uses pairwise adaptive rubrics and tournament ranking to cure "discriminative collapse" (pointwise scoring squeezes all responses into the same band), and requires running both orderings to remove position bias.
  4. Focus on the high-score tail. Chasing the Tail (arXiv 2509.21500): over-optimisation stems from being unable to tell "excellent" from "merely very good". The value of rubrics is that they can use off-policy samples while being insensitive to their surface artefacts; the key is giving rubrics the power to discriminate among several very good responses.
  5. Induce rubrics backwards from failed trajectories. The DeepVerifier approach keeps rubrics close to the real distribution of failures rather than the failures experts imagine.
  6. The self-internalization gap. A grader-free diagnostic based on the policy model's own log-probabilities that can detect when a policy trained with a weak grader stops genuinely improving. A low-cost early warning.
  7. Explicit negative items / suppressing the presence bias. Add Pitfall items, down-weight "mentioned it" items, up-weight "got it right" items, add concision constraints. The logic fits the diagnosis, but no paper has validated it systematically; it is an inference.

Specialised models are taking over the grading layer: TypeSafe Jev

Rubric grading is essentially "answer N independent closed questions about one output", which is exactly the native form of TypeSafe Jev (early access since Sep 15, 2026; currently jev-1.13.0). It takes unstructured state as input and outputs typed decisions with calibrated probabilities in one parallel pass, without generating any text. Its primitives bool / score / choice map to a rubric's met/not met, graded items and failure-mode classification. Criteria are written in natural language and need no labelled data. Pricing is $0.042/MTok input with free output, and latency is 70–500 ms. In Good Start Labs' test of 6,003 checks, Jev cost about $160 per million judgements versus about $33,000 for Claude Fable at a comparable level (GPT-5.6 Luna about $400, Gemini 3.8 Flash about $1,600). That enables a real architectural change: what used to be too expensive to do except by sampling can now be done on every single trajectory.

But its probabilities cannot be thresholded directly

An independent calibration audit (8,000 judgements × 4 phrasings × 2 base rates, compared with human labels rather than model opinions) found that Jev ranks well (AUC 0.90–0.91), but its probabilities are systematically biased toward "yes", and the rarer the positives, the worse the bias. When Jev reports about 75% confidence, only 10% of human labellers would flag the item. Raw ECE ranges from 0.157 (19.2% positive rate) to 0.209 (2.9% positive rate); a two-parameter monotonic recalibration brings ECE down to 0.023/0.007 with AUC unchanged. The same audit yields a by-product especially useful for rubric work: running four phrasings of the same question on identical state moved the mean probability from 0.445 to 0.566 and ECE from 0.397 to 0.519, while AUC barely moved. Jev can therefore expose how fragile a rubric item's wording is at very low cost.

Three more hard constraints settle its place in research evaluation: it gives no reasons (you cannot learn why an item was not met), it cannot abstain (there is no "insufficient information" state), and its context is only 64K total / 32K state, with context rot. The last rules out the most important direction from §6, process-level evaluation: BiomniBench must assess a full agent trajectory, PaperBench a whole replication artefact, DEER a long report, and none fits in 32K. Note too that the vendor's own benchmark uses the average opinion of GPT-6 Astra and Fable 5.1 as the reference answer, not human ground truth, and its tasks were written by its own team.

So its proper place is: the execution layer for leaf-level judgements (PaperBench's "judgeable in 15 minutes" rule defines exactly this kind of short, self-contained, atomic question), large-scale exhaustive scanning, routing by confidence to humans, and the architecturally different member of a cross-family grader panel. It is not suited to being the sole judge, grading trajectories or diagnosis. One judgement must be stated plainly: Jev cuts grading cost by two orders of magnitude, but doing the wrong thing cheaply is still wrong. Its probabilities lean systematically toward yes, and the known disease of rubrics is precisely inflated presence items. Wiring it uncalibrated into an RL reward loop pours fuel on reward hacking.

The ground-truth layer: how wet-lab experiments connect

A rubric can only judge whether a piece of research looks like good research, not whether its conclusion is right. In science only the latter counts. So the real question is whether the one ground truth that cannot be Goodharted, physical experiment, can enter the training loop. The answer: all three big labs use wet-lab work, but none publicly uses it as an RL reward signal.

TierApproachWho
Tier 1 · Post-hoc validation onlyWet-lab results are launch evidence, not part of the training loopOpenAI × Retro Biosciences (GPT-4b micro designed RetroSOX/RetroKLF, raising pluripotency-marker expression by more than 50×; over 30% of SOX2 suggestions beat wild type; iPSC generation cut from 3 weeks to 7 days; model not publicly released). Anthropic, Aug 2026: de novo protein binder design against 15 targets with a hit rate of 22–35% (industry norm 10–15%), validated by Adaptyv Bio + Twist. DeepMind: AI co-scientist in real labs
Tier 2 · Results fed back as training dataAI proposes hypotheses and molecules → humans test at the bench → results flow back to improve the model (the wording is "improve the model", not "use as RL reward")Anthropic has gone furthest: its own wet lab in the San Francisco Bay Area with hired bench scientists; a Life Sciences Verification Program offering up to USD 1M in Claude credits plus funding for external wet-lab validation (Modal contributing USD 250K of compute, Twist providing DNA synthesis). In essence, crowdsourced ground-truth data
Tier 3 · Wet lab as RL environmentPhysical experiments serve as the RL environment, with reality providing feedbackOnly new companies: Periodic Labs (founded by Liam Fedus, former head of post-training at OpenAI, and Ekin Dogus Cubuk, a former DeepMind GNoME author), with a 1-trillion-parameter MoE model Neon trained on a peak of 1,300 H200s; "a scientific rollout can reason and run tools for over an hour, while a training step takes only minutes", so training and inference run asynchronously. Lila Sciences' AI Science Factories are similar

Why the big labs do not do this: four structural reasons. ① The time scales differ by 3–5 orders of magnitude (training steps take minutes, experiments take days to weeks, and one RL run needs hundreds of thousands of rollouts). ② Sample efficiency and cost do not match (feeding the most expensive signal to the hungriest algorithm). ③ The signal is sparse and noisy, making credit assignment nearly impossible (a failure might come from a wrong hypothesis, or from pipetting or a reagent batch). ④ The trade-off between general and vertical models: wet-lab signal is only valid in very narrow domains, and Periodic and Lila dare to do it precisely because they gave up generality.

So frontier models actually rely on four substitutes for wet-lab work: formal verifiers (Lean, seconds, but only for math), computational simulation proxies (DFT, molecular dynamics, docking; minutes to hours, with systematic deviation from reality), retrospective ground truth (completed experiments the model has not seen; zero cost but contamination risk) and rubrics (seconds, but with the structural bias described in this section). Wet-lab work itself serves as final endorsement and calibration, not as a source of gradients.

The core architectural judgement of this piece

Research benchmarks with wet-lab ground truth are necessarily small, slow and expensive (BioLab Bench and BiomniBench are both in the hundred-task range) and can never reach the scale of HLE, so their role should not be a leaderboard but a calibrator. The future of research evaluation is not one benchmark but two layers: a large, cheap rubric layer for day-to-day iteration, and a small, expensive ground-truth layer that periodically calibrates the first. Whoever controls the second defines how credible the first is. Today almost everything in both the US and China is the first layer, and nobody builds the calibration layer in between (regressing rubric weights on ground-truth outcomes). UniScientist uses uniform weights, RaR uses hand-picked 5 / 3–4 / 1–2 tiers, and HealthBench uses physicians' importance judgements; none of the three calibrates against outcomes.

What to watch: when Periodic Labs reports its first new material or superconductivity result obtained through a closed RL loop (rather than human intuition plus AI assistance). That is the only hard test of whether the Tier 3 route works. Its Sep 2026 technical post covered only training and serving infrastructure and explicitly reported no newly validated materials or superconductivity results.

8. China's newcomers: UniPat, Humanlaya and Zhiyan Huisheng

China's "AI evaluation plus high-quality training data" companies fall roughly into three tiers. The sector only really got going in the second half of 2025, and most of the leaders are less than a year old. The first tier is the true counterpart to Mercor, AfterQuery and Turing in the US.

DateEventDetail
Late 2025UniPat AI foundedFounder Kuan Li (Li Kuan), formerly of Alibaba's Tongyi AI Lab; focus on post-training analysis, data synthesis and RL
2025Humanlaya (formerly Rang Zhineng) foundedHaidian, Beijing; an AI data lab bringing together PhDs and research experts
Jan 2026Zhiyan Huisheng foundedLegal representative Zhu Xizhou; very little public information
Jun 2026Tencent (via Shanghai Qishan Investment) invests in Zhiyan Huisheng–
Early Sep 2026Humanlaya closes a Pre-A round of several hundred million RMBLed by CDH Investments (Hong Kong fund), with HongShan, Capital Today and BAI Capital participating
Sep 2026UniPat AI closes a USD 300M Series ALed by Alibaba, with Tencent and HongShan participating; valued at about USD 2.5B, the largest deal in the sector

The tiers

Tier 1: new-generation evaluation and expert-data companies

  • UniPat AI: coding, browser and multimodal visual-reasoning evaluations; the Terminal-X suite (DeepTerminalBench / EvoCode-Bench / RoadmapBench, all built on the Terminal-Bench 2.0 format + Harbor) plus Monthly-SWEBench, SaaS-Bench and ClawBench; and it trains its own models (UniScientist-30B-A3B, UniMath-35B-A3B), see below
  • Humanlaya: One-Million Bench / BiomniBench; high-end expert annotation across medicine, finance, law, manufacturing and education
  • Zhiyan Huisheng: AI industry application systems integration; business details unverified
Direct counterparts to Mercor, AfterQuery and Turing (whose ARR in the US already exceeds USD 100M, with regular work for Ant, Alibaba and ByteDance).

Tier 2: traditional annotation firms upgrading

  • Haitian Ruisheng (688787.SH): listed on the STAR Market; 2025 revenue RMB 377M (+59%)
  • Datatang, Testin Data, DataBaker, Zhengshu Zhineng, TalkingData
Mainly sell annotation hours and datasets; trying to move toward expert data and RL environments, but their DNA is closer to "selling human time".
HongShan's dense positioning

UniPat, Humanlaya and Zhiyan Huisheng all have HongShan involvement, on top of HongShan's own incubated xbench evaluation system (two tracks: a capability-ceiling track and a real-world-value track; "evergreen evaluation" updated dynamically; it has released xbench-ScienceQA, DeepSearch and AgentIF, and BabyVision jointly with UniPat). HongShan is the most densely positioned investor in the "evaluation defines model value" sector.

UniPat is not just an evaluation company: it trains its own models (added Sep 20, 2026)

Earlier versions missed this, and it changes how the company should be characterised. Besides the benchmarks catalogued in §4, UniPat's GitHub organisation (UniPat-AI, 9 public repos) holds three non-benchmark products:

ProductSpecMethodSelf-reported results
UniScientist-30B-A3B
Mar 2026, research-agent model
Qwen3-MoE architecture, 30B total / 3B active, Apache-2.0; blog only, paper marked "Coming Soon"Evolving Polymathic Synthesis: an LLM acts as cross-disciplinary question writer and human experts as high-precision verifiers; rubrics pass three filters (objective consistency, discriminability, atomicity). Training = agentic SFT + a Report Aggregation objective (synthesising a final report from N candidates, with references produced by rubric-based rejection sampling). Dataset: 4,700+ research-grade samples, 20+ rubric items per question, 50+ disciplines, 1–2 hours of expert time per sampleFrontierScience-Research 28.3 (33.3 at Aggr@8) vs GPT-5.2 25.2 / Claude Opus 4.5 17.5; DeepResearch Bench II 48.0 vs OpenAI DR 45.4. Full reasoning trajectories released (far more responsible than releasing scores alone); training data not released
UniMath-35B-A3B
Jul 2026, olympiad-level proof model
Base explicitly Qwen/Qwen3.6-35B-A3B, 35B total / 3B active, Apache-2.0; also blog onlyTraining data supervises the revision process of a proof: initial draft (exposing obligations to prove) → self-assessment (a "proof debt ledger" naming the first real obstacle) → targeted repair or a change of route → final synthesis. At inference, pool_refine_tournament: sample, self-assess and rank → repair the strongest imperfect candidate → merge survivors pairwise, keeping only measurable improvementsIMO 2025 35/42, USAMO 2026 36/42, IMO-ProofBench 86.0% (without TTS: 19.75 / 17.88 / 52.6%)
SWE-VisionAgentic VLM framework (stateful Jupyter inside Docker)A tool, not a benchmark–
The UniMath numbers need two discounts

① The with-TTS and without-TTS scores use different grading. The former is avg@8 graded by a single judge, DeepSeek-V4 Pro; the latter is the final proof graded by three judges (Gemini 3.1 Pro + GPT-5.5 (xhigh) + DeepSeek-V4 Pro) plus human expert review. How much of the jump from 19.75 to 35 comes from TTS and how much from the change in grading cannot be separated from public information, which is once again the theme of §5. ② "Gold medal" is relative to the IMO cut-off, not to SOTA. On IMO-ProofBench it ranks fourth (GPT-5.6-Sol at 91.1% and Claude Fable 5 at 89.0% are both above it). The real story is lifting a base model from 23.1% to 86.0%: an impressive gain, but still below frontier closed models. Also, the blog text says "IMO 2026" while the table says IMO 2025; when citing, go by the table.

Team lineage: UniScientist is essentially a sequel to Tongyi DeepResearch

UniScientist's citation lists 9 authors with affiliations 1 UniPat AI, 2 Peking University (no Alibaba affiliation). But Kuan Li is the founder, and Baixuan Li and Jialong Wu were, with him, core authors of the Tongyi DeepResearch technical report and WebSailor (Tongyi Lab, Alibaba Group). The methodological lineage is plain: long-horizon information-seeking agents, tool loops and rollout aggregation all extend the WebSailor / Tongyi DeepResearch line.

Two easily confused points need stating clearly. ① The two models are not "made by Tongyi"; they are built on open Qwen weights (UniMath's HF metadata states its base_model explicitly; UniScientist's architecture tag is qwen3_moe, and the only community discussion on its HF page is one reminding it that it does not name its base). ② Whether all these authors are on UniPat's payroll cannot be confirmed; some may still be at Tongyi, in a cross-institution collaboration.

But this relationship explains why Alibaba led a USD 300M round. Chinese media described it as "Alibaba to lead investment in former employee's AI evaluation company", and a Sina headline was blunter: "backing a former intern's startup". For Alibaba it achieves three things at once: retaining talent, free ecosystem endorsement (all of UniPat's models are built on Qwen), and locking in an evaluation-data supplier.

A conflict of interest the §9 framework does not yet cover

The question-setter is also a contestant, and the referee's technical taste has an origin. UniPat is described as a third-party evaluator, but its base models come from Alibaba, its money comes from Alibaba and its team comes from Alibaba, and its leaderboards are grading Qwen 3.6 Plus and its competitors. No cheating is needed for bias: a research or agent evaluation designed by people from Tongyi DeepResearch will naturally reward that style of solution (long-horizon search, tool loops, rollout aggregation), which is exactly what Qwen-family models are good at. This is a different kind of problem from the US pattern of "customers citing their own supplier", and harder to spot.

It is not yet a closed loop: its own benchmarks cover coding, GUI and vision, while its models target research and math, so the two do not overlap. The real risk point is the day it uses the same Evolving Polymathic Synthesis pipeline to produce a research benchmark. Training and test sets would then be products of the same distribution. That is not cheating, but the evaluation would lose its information value.

Cold data on adoption (API measurements, Sep 20, 2026)

Hugging Face downloads: UniScientist 75 (since its March release), UniMath 18 (since July), with 15 and 2 likes; 1 and 0 HF discussion threads, and that single thread is the one pointing out the missing base model. No posts found on Reddit (including r/LocalLLaMA) or Hacker News. There is Chinese coverage, but the same article, "Can AI really do research…", appeared in Sina Finance's "company news" section, PingWest, TMTPost and a Zhihu column, which is PR distribution; the only genuinely independent editorial coverage found was from QbitAI.

A 30B / 3B-active model is the size the local-deployment community should love most, yet nobody runs it: no GGUF quantisation, no hands-on posts. The reasonable characterisation is a "fundraising narrative asset" rather than an "open-source model", serving as technical backing for a USD 2.5B valuation. As a business that makes complete sense: UniPat's customers are model makers and enterprises, 75 downloads do not affect its data sales, and one QbitAI article may be worth more than ten thousand downloads.

Team background checks

UniPat's founder is public (Kuan Li, formerly of Alibaba's Tongyi AI Lab); it is the only one of the three that actively discloses its founder and uses big-tech technical background as endorsement. The founder of Humanlaya (Rang Zhineng) has not been named in any public report; the website has no team page and the GitHub organisation's member list is private, so only the investor line-up can be confirmed. For Zhiyan Huisheng, public business-registry records show only the legal representative; there is no public reporting on team or product, making it the lowest-profile of the three.

Citation: the "external adoption" step has not happened yet

We found no evidence that UniPat, xbench or Humanlaya benchmarks are formally cited in the official technical reports of DeepSeek, Qwen, Zhipu GLM, Kimi, MiniMax or other leading model makers. Current "citations" happen mainly among insiders (UniPat and xbench jointly releasing BabyVision is essentially one ecosystem endorsing itself). By contrast, C-Eval, CLUEWSC, CMMLU and OpenCompass are formally cited in the DeepSeek-V3 and Qwen technical reports. These new companies' credibility currently shows up in funding markets (Alibaba and Tencent leading, a USD 2.5B valuation) rather than in technical-report citations, two entirely different states of trust.

A recheck on Sep 20, 2026 confirmed this and allows a finer grading. Only four kinds of "citation" turned up, and the first three are not external adoption. ① UniPat evaluating others: the SaaS-Bench and RoadmapBench leaderboards include Claude Opus 4.7, GPT-5.4, Gemini 3.1 Pro, DeepSeek V4 Pro, GLM 5.1, Kimi K2.6, Qwen 3.6 Plus and MiniMax M2.7, and the BabyVision board includes Gemini3-Pro-Preview, GPT-5.2, Doubao-Seed-1.8 and Qwen3VL-235B. The direction is reversed. ② Reposting by third-party aggregators (BenchmarkList has leaderboard pages for RoadmapBench and BabyVision), which counts as half a step. ③ Mutual endorsement within the ecosystem (BabyVision = xbench × UniPat, and HongShan invested in both). ④ The real counter-evidence: the Qwen3-Coder-Next technical report (Mar 2026) uses SWE-Bench Verified / Multilingual / Pro + TerminalBench 2.0, and none of UniPat's benchmarks.

9. Cited is not trusted: three layers of credibility, and where to look

A benchmark's credibility is not one-dimensional. It can be split into three layers, and the US and China are at different levels of maturity in each:

LayerUSChina
① Cross-vendor common standards
Who maintains them; are they in every lab's model card?
MMLU / GPQA / SWE-bench Verified: academic origins, adopted consistently by OpenAI, Anthropic, Google and xAIC-Eval / CLUE / OpenCompass / CMMLU: originated at universities and national research institutes, formally cited in the DeepSeek-V3 and Qwen technical reports; firmly established
② Vendor-built leaderboards
Adopted by third parties beyond their own customers?
SEAL / APEX-Agents / GDP.pdf / Riemann-bench are cited in Anthropic and OpenAI system cards, but almost always as "customers citing their own supplier", with obvious selectivityUniPat / xbench / Humanlaya: no evidence yet of citation in third-party model makers' technical reports; credibility shows up in funding markets rather than technical reports
③ Public / media popularity boards
Horizontal comparison for the public, not academic or commercial adoption
Chatbot Arena / LMArena: once the most credible, weakened in 2025–26 by disputes over "preference optimisation" vote gamingSuperCLUE, Feixian Zhineng's ReLE: track mainstream model rankings continuously; credibility concentrated at the consumer-awareness level

The two regions are structurally the same: what is truly used across vendors is always the non-commercial academic standard, and vendor leaderboards at most fill niches and serve as sales endorsements. The difference is maturity. US vendor leaderboards have made a small entry into top labs' official system cards, while the Chinese newcomers that started in 2025 still earn trust from capital markets.

A signal that the sector has entered a new stage

When a third-party model maker that is not a customer cites a UniPat, Humanlaya or xbench leaderboard in its technical report, that will be the substantive signal that China's vendor-evaluation sector has moved from "fundraising narrative" to "industry adoption". It has not been observed yet.

Where to find more benchmarks

Agent-specific, most rigorous: HAL, the Holistic Agent Leaderboard (hal.cs.princeton.edu). Built by Princeton's SAgE team with an ICLR 2026 paper. It includes only 9 genuinely multi-step agentic benchmarks (no pure-knowledge tests like MMLU), and its core value is recording both the model and the scaffold, scoring each model × scaffold combination separately: 9 models × 9 benchmarks, 21,730 rollouts and about USD 40,000 in validation cost. It has stopped accepting new submissions and pivoted to "agent reliability" research (consistency, predictability, robustness, safety, refusal rate). Its academic credibility is solid (Princeton + ICLR, with API credits from OpenAI), but no leading lab has yet formally cited it in a system card.

Live aggregatorNotes
Artificial AnalysisCompares models on intelligence, speed and price
OpenRouter BenchmarksBased on real API traffic; leans toward agent and generation scenarios
llm-stats.comRankings of 300+ models
BenchLM.aiTracks 439 benchmarks in eight categories
Epoch AI BenchmarksCollects third-party results and runs its own internal evaluations
AIMultipleClaims 800+; widest coverage but uneven depth
Static curated list on GitHubNotes
philschmid/ai-agent-benchmark-compendium50+ agent benchmarks; the clearest four-way classification
panilya/awesome-ai-benchmarks100+ benchmarks; broad coverage
supernalintelligence/Awesome-General-Agents-BenchmarkFocused on general agent capability
benchflow-ai/awesome-evalsA "non-BS" curated list

10. Nine takeaways

  1. Benchmarks are the sales tentacles of an $8.5 billion training-data and RL-environment business, not standalone products. The grader (verifier) is the most expensive and central part of the chain, and that decides who has a motive to build benchmarks and who does not.
  2. Vendor leaderboards are sales funnels; academic leaderboards are public infrastructure. The first measures success by customers converted, the second by citations in papers and system cards. The US and China are structurally the same; China's gap is in maturity, not structure.
  3. Nearly half of benchmarks need a dedicated harness, mostly proprietary and unshared. Harbor is the rare exception, adopted by several vendors of different backgrounds.
  4. Harness and scaffold choice often moves scores more than the gap in model ability (up to 48 percentage points). It is now a research direction in its own right (Harness-Bench and others), not a fluke.
  5. Research benchmarks have gone from scarce to crowded but fragmented. At least 7–9 new research-agent benchmarks appeared in H1 2026 (from Meta, Microsoft, EPFL, the Shanghai AI Laboratory and others), but none has become a cross-lab standard written into system cards. The bottleneck has shifted from "nobody writes the questions" to "nobody agrees", and the openness and rubric subjectivity of research tasks make the harness sensitivity of §5 even more extreme than in coding (GAIA: the same 466 questions scored anywhere from 44.8% to 92.36% across leaderboards).
  6. Cited is not trusted; credibility must be read in three layers: common standards, vendor leaderboards and popularity boards. The US and China differ in maturity at each layer, so calling any leaderboard simply "authoritative" is inaccurate.
  7. China's new "evaluation-first" companies have not yet been adopted in third-party model cards. Their credibility comes from funding markets (Alibaba, Tencent, HongShan), not technical-report citations, and that is the key thing to watch. UniPat is no longer a pure evaluator (it trains UniScientist and UniMath), and "the question-setter is also a contestant, and the referee's taste has an origin" is a new kind of conflict of interest added in §8, harder to spot than the US pattern of customers citing their own supplier.
  8. The US and China have converged on the same method: rubrics. Open-ended tasks are broken into N independently judgeable items. The cost is quietly swapping the target from "right" to "complete": after rubric RL, presence items rise while factual correctness, concision, relevance and overall quality all fall, and rubric-free judges even prefer the pre-training base model. The gap is not grading precision (single-item judgement already matches human expert agreement) but the assumption that adding up N items equals quality. So for any rubric leaderboard, the first question is the grader's agreement with human experts, and only four of ten publish it.
  9. Only external ground truth can correct this substitution, and wet-lab experiments are not currently in any frontier model's RL loop. The three big labs use wet-lab work for post-hoc validation and data feedback (Anthropic goes furthest, with its own wet lab and funding for external validation). Only new companies that gave up generality, such as Periodic Labs and Lila, treat physical experiments as an RL environment, because the time scales differ by 3–5 orders of magnitude. Research benchmarks with ground truth are therefore necessarily small, slow and expensive, and their role should be calibrator, not leaderboard: a large, cheap rubric layer for daily iteration and a small, expensive ground-truth layer that periodically calibrates it. The calibration layer in between (regressing rubric weights on ground truth) has not been built by anyone.

Adapted from the Spark Science research notes (originally published in Chinese at jimmyresearch.com). Last updated Sep 20, 2026.