Why a separate column on LLM data
The problem in robotics data is that every demonstration needs a person, equipment and a site to produce it. Large language models are the opposite: pretraining corpora come almost entirely from free public web crawls, at close to zero marginal cost. So both the value and the controversy have moved away from collection to other places: the compute for cleaning and deduplication, the legal boundaries of copyright, expert hours in post-training, the automatic grader inside an RL environment, and the conversations of 900 million weekly active users that flow back every day after a product ships.
The leading recipes of 2026 already show the structure. Pretraining runs to tens of trillions of tokens with sources largely undisclosed. Post-training data is only hundreds of thousands to a few million examples, yet it determines a model's style and capabilities. And the most expensive layer, RL environments, is becoming something labs build in-house, with outsourcing taking only the long tail. Who supplies this data, how models mix it, what a million tokens costs, and which legal line it runs into: those are the questions this column answers.
The six-layer data chain
The layers are ordered by production method and by distance from the model's final behaviour. The usual "pretraining / post-training" split is broken into six layers here, because the three post-training routes (human annotation, RL environments, synthesis and distillation) have completely different cost structures, and because copyright licensing and deployment feedback each became a market and a priced object in their own right during 2025–2026.
| Layer | Data | How it is produced | Typical scale | Role in the model |
|---|---|---|---|---|
| 1 | Pretraining corpora | Public crawl → filter → dedup → mix; upstream is almost entirely Common Crawl | 10T–40T tokens (Qwen3: 36T); open corpora: FineWeb (Hugging Face) launched at 15T (now about 18.5T), Dolma 3 (Ai2) three pools totalling 9.3T | Largest volume, lowest cost; filtering-based recipes discard about 90% of raw data |
| 2 | Copyright licensing and data deals | Turning existing content into rights: annual fees, per book, per image, per request | $50K to $50M per year per deal; Bartz v. Anthropic settlement about $3,100 per work | Lowers litigation risk, buys exclusivity and real-time access; token-based pricing almost never appears |
| 3 | Human post-training data | Experts write prompts, grade, compare preferences; labs write the spec, vendors produce | SFT: hundreds of thousands to a million examples; preference pairs in the hundreds of thousands | Alignment layer: few examples, heavy weight; now a $50–250/h expert labour market |
| 4 | RL environments and verifiable rewards | Interactive world + tasks + automatic grader + resettable infrastructure | Tülu 3 RLVR uses only 29,946 examples; DeepSeek V3.2 synthesised 1,827 environments | Rate-limiter on agent capability; the most expensive part is the grader, not the container |
| 5 | Synthetic data and distillation | Rewriting real documents / teacher generation / reasoning-trace distillation / AI feedback | Phi-4 pretraining about 55% synthetic or rewritten; Nemotron-CC (NVIDIA): about 1.9T of 6.3T is rewritten | An amplifier; the cost is in verification and filtering, not generation |
| 6 | Deployment feedback and user data | Consumer products train by default with opt-out; business / API tiers do not train by default | ChatGPT 900M weekly actives (company figure); nobody discloses the share used | Raw material for post-training and evals; the cost is compliance and trust, not collection |
Chapter 10, Data recipes and suppliers by model company, breaks down more than 40 Chinese and US model companies layer by layer (which layers they use, how many tokens, built or bought, and from whom), with a company × data-layer heatmap and a supplier matrix.
Three cross-cutting chapters complete the column:
- Open datasets, formats and tooling: the Hugging Face Hub, four open processing pipelines such as datatrove, tokenizers and decontamination.
- Vendors, litigation and policy: a lab procurement market of 50+ data vendors worth about $8.5B a year; four lines of case law (Bartz, NYT, Kadrey, Ross); diverging US, EU and Chinese policy; crawler governance.
- Prices, costs and mixtures: a price band spanning six orders of magnitude, and every model with a public data mixture listed one by one.
Implied price per million tokens
Implied price of each data layer, USD per million tokens (log scale)
Data priced by the hour, by the book or by the task is converted into dollars per million tokens and placed on the same axis as synthetic tokens. The range exceeds five orders of magnitude: human data is several orders of magnitude more expensive than machine-generated tokens.
Data table
| Layer / type | Raw price (low–high) | Unit | Implied $/M tokens | Basis |
|---|---|---|---|---|
| Synthetic tokens: commercial API output | 0.6–10 | $/M tokens | 0.6–10 | DeepSeek's official page: deepseek-flash off-peak output $0.6 (peak $1.2), up to GPT-4o at $10 per million output tokens; GPT-4o price from a third-party aggregator, accessed Oct 15, 2026 |
| Synthetic tokens: self-hosted open model | 0.055–0.55 | $/M tokens | 0.055–0.55 | Author's estimate: single H100 at 1k–10k tokens/s, $2/h; low confidence |
| User data: Muse contributor discount | 1.15–4.05 | $/M tokens | 1.15–4.05 | Input/output price gap between the standard tier and the contributor tier; to be verified |
| General annotation (incl. secondhand self-reports) | 4–40 | $/h | 2,000–20,000 | Remotasks self-reported $4–7 up to $40 for Outlier on an aggregator site; secondhand, low reliability |
| Chinese expert annotation | 14–55 | $/h | 7,000–27,500 | RMB 100–400/h converted at about 7.3; media figure |
| US expert annotation (STEM / finance) | 55–100 | $/h | 27,500–50,000 | Rates listed on Mercor job pages, excluding the platform's cut |
| RL environment: per task | 200–2,000 | $/task | 2,000–20,000 | Epoch interviews: respondents said mostly $200–2,000; complex software-engineering tasks at about $20K are rare |
| Book settlement: Bartz v. Anthropic | 3,000 | $/book | 30,000 | $1.5B / about 480K works (482,460 on the list; 440,490 qualifying at final approval); includes compensation for pirated acquisition, so not a pure training-licence price |
| Book licence: HarperCollins | 5,000 | $/book | 50,000 | $5,000 per backlist title for 3 years; buyer reportedly Microsoft (anonymous sources) |
Key findings
- Pretraining corpora are a public good; competition is in the filter, not the source. The open corpora FineWeb (Hugging Face), DCLM (DataComp-LM), Dolma (Ai2) and Nemotron-CC (NVIDIA) are all different filterings, deduplications or rewrites of Common Crawl. Common Crawl itself is a non-profit of about 18 people; its Aug 2026 crawl covered about 2.14 billion pages, and about 75% of top news sites now block CCBot. The responses to native tokens running out are model-based quality filtering, additional PDF and multilingual data, and synthetic rewriting to fill the gap.
- Copyright now has a price anchor, but it is a damages price, not a licence price. Bartz v. Anthropic settled for $1.5B over about 482,460 works, roughly $3,100 per work (440,490 works qualified at final approval). At an assumed 100,000 tokens per book, that is about $31,000 per million tokens, three to four orders of magnitude above API output prices. The court's split was "training is fair use, pirated acquisition is not" (Judge Alsup), while Kadrey v. Meta left open a "market dilution" path. The licensing market is shifting from one-off training licences towards ongoing access and pay-per-request.
- The core assets in human post-training data are neutrality and expert density. After Meta paid $14.3B for 49% of Scale AI, Google, OpenAI and xAI cut their orders, which flowed to Surge AI, Mercor and Handshake AI. Mercor roughly tripled in a year to an annualised gross revenue of about $2B (this includes payouts to experts and cannot be compared directly with Surge's or Scale's net revenue). Crowdsourced labelling has been pushed to the margins; expert prompt-writing and grading at $50–250/h is now the mainstream.
- RL environments are the most expensive layer in 2026, and the one labs are internalising fastest. A single task costs $200–2,000; a high-fidelity replica of a software product about $300K. Google paid about $1.5B to bring in the Mechanize team rather than expanding its Scale contract (media figure from a single Business Insider report; not officially disclosed; structured as a licence plus hiring). Chinese frontier labs build almost all their environments in-house: DeepSeek, Kimi (Moonshot AI) and Zhipu (Z.ai) all report environment or tool counts in the tens of thousands in their technical reports. Third-party providers (Prime Intellect and some 35–42 others) are mostly small seed-stage teams.
- Synthetic data is now a structural ingredient, and the debate has moved from "will it collapse" to "is distillation allowed". Industrial practice (faithful rewriting, mixing with real data, verification and filtering) is equivalent to the "accumulate + verify" route that academic work has shown does not collapse. In Feb 2026 Anthropic alleged that about 24,000 fraudulent accounts made more than 16 million interactions for distillation (by lab: MiniMax 13M+, Moonshot 3.4M+, DeepSeek 150K+). This is Anthropic's one-sided account with no judicial finding, and the legal basis is terms of service, not copyright.
- Recipe openness determines how much detail this column can give. Only Ai2 (OLMo / Dolma / Tülu), Swiss AI and Hugging Face publish complete recipes. Llama 3 gives broad category shares (50/25/17/8). Qwen (Alibaba), DeepSeek and Kimi give only total tokens and stage structure. Mistral, Gemma and Llama 4 disclose essentially nothing. None of the primary documents from 23 Chinese model companies names a specific supplier such as Speechocean (Haitian Ruisheng) or Datatang, and the suppliers' annual reports are equally anonymous: the blank itself is the finding.
- Unlike robotics data, LLM data can be measured on a single ruler. Every layer can be converted into an implied per-token price, from $0 (open corpora) to about $31,000 per million tokens (the book settlement, assuming 100,000 tokens per book), spanning six orders of magnitude. Robotics data can only be priced by the hour, and its layers are not interchangeable.
Sourcing conventions
Figures from technical reports and papers are marked as paper figures; companies' own statements as company figures; media reports as media figures; extrapolations from public unit prices as the author's estimates; anything that could not be traced to a primary source is flagged ⚠️. Each chapter ends with a table of conflicting and unverified claims, listing circulating figures that were refuted or remain doubtful, for example "only about 40% of licences in 2026 cover training use", "Reddit–OpenAI at $70M a year", and "DCLM 7B trained on 4T tokens" (it was actually trained on 2.6T; 4T is the dataset size). These claims are not used as facts in the main text.
Chapters
| # | Chapter | What it covers |
|---|---|---|
| 01 | Pretraining corpora | Open pretraining corpora almost all derive from Common Crawl; the competition has moved from rule-based filtering to model scoring and synthetic rewriting. |
| 02 | Copyright licensing and data deals | AI licensing prices for news, communities, books and image libraries; intermediary marketplaces and pay-per-crawl; the Bartz settlement as a price anchor. |
| 03 | Human post-training data | SFT demonstrations, preference comparisons, expert question banks and rubrics: from crowdsourced piecework to a $50–250/h expert market. |
| 04 | RL environments and verifiable rewards | With post-training shifting to RL, the supply of verifiable tasks limits agent capability: how environments are built, who sells them and at what price. |
| 05 | Synthetic data and distillation | Textbook synthesis, web rewriting, instruction synthesis, reasoning distillation and AI feedback: who uses them, in what share, at what cost. |
| 06 | Deployment feedback and user data | How conversations, feedback and agent trajectories flow back into training: each product's training policy, data-for-discount schemes and compliance boundaries. |
| 07 | Open datasets, formats and tooling | Hosting and formats, processing toolchains, tokenizers, open post-training sets, decontamination and licence grey zones. |
| 08 | Vendors, litigation and policy | Market structure of data vendors, funding and M&A, the gradient of copyright cases, national policies and crawler governance. |
| 09 | Prices, costs and mixtures | The price band from $0 open corpora to $300K environments, and how much open models disclose about their data mixtures. |
| 10 | Data recipes and suppliers by model company | How about 50 model companies in China, the US and elsewhere mix the six layers, whom they buy from, and who supplies whom. |
The column also maintains 164 entity cards: 83 companies and institutions, 27 datasets, 7 tools and formats, 34 concepts and methods, and 13 papers and benchmarks. These deep pages have not been migrated yet.
This column is a personal research compilation. Every figure is labelled in the text with its source and basis; companies' self-reported data is marked as a company figure, and anything not independently verified is flagged ⚠️. All judgements are as of the date of writing and are not investment or R&D advice.
Adapted from the Spark Science research notes (originally published in Chinese at jimmyresearch.com). Last updated Oct 15, 2026.