AI Data · LLM

LLM Data

A six-layer data chain for large language models: pretraining corpora, copyright licensing, human post-training data, RL environments, synthesis and distillation, and deployment feedback. Who supplies it, how models mix it, what it costs, and which legal lines constrain it.

Spark Science ResearchUpdated Oct 15, 20269 min read
10
chapters
164
entity cards
6
data layers
40+
model companies mapped

Why a separate column on LLM data

The problem in robotics data is that every demonstration needs a person, equipment and a site to produce it. Large language models are the opposite: pretraining corpora come almost entirely from free public web crawls, at close to zero marginal cost. So both the value and the controversy have moved away from collection to other places: the compute for cleaning and deduplication, the legal boundaries of copyright, expert hours in post-training, the automatic grader inside an RL environment, and the conversations of 900 million weekly active users that flow back every day after a product ships.

The leading recipes of 2026 already show the structure. Pretraining runs to tens of trillions of tokens with sources largely undisclosed. Post-training data is only hundreds of thousands to a few million examples, yet it determines a model's style and capabilities. And the most expensive layer, RL environments, is becoming something labs build in-house, with outsourcing taking only the long tail. Who supplies this data, how models mix it, what a million tokens costs, and which legal line it runs into: those are the questions this column answers.

The six-layer data chain

The layers are ordered by production method and by distance from the model's final behaviour. The usual "pretraining / post-training" split is broken into six layers here, because the three post-training routes (human annotation, RL environments, synthesis and distillation) have completely different cost structures, and because copyright licensing and deployment feedback each became a market and a priced object in their own right during 2025–2026.

LayerDataHow it is producedTypical scaleRole in the model
1Pretraining corporaPublic crawl → filter → dedup → mix; upstream is almost entirely Common Crawl10T–40T tokens (Qwen3: 36T); open corpora: FineWeb (Hugging Face) launched at 15T (now about 18.5T), Dolma 3 (Ai2) three pools totalling 9.3TLargest volume, lowest cost; filtering-based recipes discard about 90% of raw data
2Copyright licensing and data dealsTurning existing content into rights: annual fees, per book, per image, per request$50K to $50M per year per deal; Bartz v. Anthropic settlement about $3,100 per workLowers litigation risk, buys exclusivity and real-time access; token-based pricing almost never appears
3Human post-training dataExperts write prompts, grade, compare preferences; labs write the spec, vendors produceSFT: hundreds of thousands to a million examples; preference pairs in the hundreds of thousandsAlignment layer: few examples, heavy weight; now a $50–250/h expert labour market
4RL environments and verifiable rewardsInteractive world + tasks + automatic grader + resettable infrastructureTülu 3 RLVR uses only 29,946 examples; DeepSeek V3.2 synthesised 1,827 environmentsRate-limiter on agent capability; the most expensive part is the grader, not the container
5Synthetic data and distillationRewriting real documents / teacher generation / reasoning-trace distillation / AI feedbackPhi-4 pretraining about 55% synthetic or rewritten; Nemotron-CC (NVIDIA): about 1.9T of 6.3T is rewrittenAn amplifier; the cost is in verification and filtering, not generation
6Deployment feedback and user dataConsumer products train by default with opt-out; business / API tiers do not train by defaultChatGPT 900M weekly actives (company figure); nobody discloses the share usedRaw material for post-training and evals; the cost is compliance and trust, not collection

Chapter 10, Data recipes and suppliers by model company, breaks down more than 40 Chinese and US model companies layer by layer (which layers they use, how many tokens, built or bought, and from whom), with a company × data-layer heatmap and a supplier matrix.

Three cross-cutting chapters complete the column:

  • Open datasets, formats and tooling: the Hugging Face Hub, four open processing pipelines such as datatrove, tokenizers and decontamination.
  • Vendors, litigation and policy: a lab procurement market of 50+ data vendors worth about $8.5B a year; four lines of case law (Bartz, NYT, Kadrey, Ross); diverging US, EU and Chinese policy; crawler governance.
  • Prices, costs and mixtures: a price band spanning six orders of magnitude, and every model with a public data mixture listed one by one.

Implied price per million tokens

Implied price of each data layer, USD per million tokens (log scale)

Data priced by the hour, by the book or by the task is converted into dollars per million tokens and placed on the same axis as synthetic tokens. The range exceeds five orders of magnitude: human data is several orders of magnitude more expensive than machine-generated tokens.

$0.01$0.1$1$10$100$1,000$10,000$100,000Synthetic: commercial API output$0.6–$10Synthetic: self-hosted open model$0.055–$0.55User data: Muse contributor discount$1.15–$4.05General annotation (incl. self-reports)$2,000–$20,000Chinese expert annotation$7,000–$27,500US expert annotation (STEM/finance)$27,500–$50,000RL environment: per task$2,000–$20,000Book settlement: Bartz v. Anthropic≈$30,000Book licence: HarperCollins≈$50,000
Conversion assumptions: 100,000 tokens per book; 2,000 tokens of output per hour for expert and general annotation; 100,000 tokens per RL environment task; about RMB 7.3 per USD. Synthetic and user data are already priced per token and are not converted. Open corpora cost $0 and cannot be shown on a log axis. The conversion assumptions are the author's own; read the chart for orders of magnitude only.
Data table
Layer / typeRaw price (low–high)UnitImplied $/M tokensBasis
Synthetic tokens: commercial API output0.6–10$/M tokens0.6–10DeepSeek's official page: deepseek-flash off-peak output $0.6 (peak $1.2), up to GPT-4o at $10 per million output tokens; GPT-4o price from a third-party aggregator, accessed Oct 15, 2026
Synthetic tokens: self-hosted open model0.055–0.55$/M tokens0.055–0.55Author's estimate: single H100 at 1k–10k tokens/s, $2/h; low confidence
User data: Muse contributor discount1.15–4.05$/M tokens1.15–4.05Input/output price gap between the standard tier and the contributor tier; to be verified
General annotation (incl. secondhand self-reports)4–40$/h2,000–20,000Remotasks self-reported $4–7 up to $40 for Outlier on an aggregator site; secondhand, low reliability
Chinese expert annotation14–55$/h7,000–27,500RMB 100–400/h converted at about 7.3; media figure
US expert annotation (STEM / finance)55–100$/h27,500–50,000Rates listed on Mercor job pages, excluding the platform's cut
RL environment: per task200–2,000$/task2,000–20,000Epoch interviews: respondents said mostly $200–2,000; complex software-engineering tasks at about $20K are rare
Book settlement: Bartz v. Anthropic3,000$/book30,000$1.5B / about 480K works (482,460 on the list; 440,490 qualifying at final approval); includes compensation for pirated acquisition, so not a pure training-licence price
Book licence: HarperCollins5,000$/book50,000$5,000 per backlist title for 3 years; buyer reportedly Microsoft (anonymous sources)

Key findings

  1. Pretraining corpora are a public good; competition is in the filter, not the source. The open corpora FineWeb (Hugging Face), DCLM (DataComp-LM), Dolma (Ai2) and Nemotron-CC (NVIDIA) are all different filterings, deduplications or rewrites of Common Crawl. Common Crawl itself is a non-profit of about 18 people; its Aug 2026 crawl covered about 2.14 billion pages, and about 75% of top news sites now block CCBot. The responses to native tokens running out are model-based quality filtering, additional PDF and multilingual data, and synthetic rewriting to fill the gap.
  2. Copyright now has a price anchor, but it is a damages price, not a licence price. Bartz v. Anthropic settled for $1.5B over about 482,460 works, roughly $3,100 per work (440,490 works qualified at final approval). At an assumed 100,000 tokens per book, that is about $31,000 per million tokens, three to four orders of magnitude above API output prices. The court's split was "training is fair use, pirated acquisition is not" (Judge Alsup), while Kadrey v. Meta left open a "market dilution" path. The licensing market is shifting from one-off training licences towards ongoing access and pay-per-request.
  3. The core assets in human post-training data are neutrality and expert density. After Meta paid $14.3B for 49% of Scale AI, Google, OpenAI and xAI cut their orders, which flowed to Surge AI, Mercor and Handshake AI. Mercor roughly tripled in a year to an annualised gross revenue of about $2B (this includes payouts to experts and cannot be compared directly with Surge's or Scale's net revenue). Crowdsourced labelling has been pushed to the margins; expert prompt-writing and grading at $50–250/h is now the mainstream.
  4. RL environments are the most expensive layer in 2026, and the one labs are internalising fastest. A single task costs $200–2,000; a high-fidelity replica of a software product about $300K. Google paid about $1.5B to bring in the Mechanize team rather than expanding its Scale contract (media figure from a single Business Insider report; not officially disclosed; structured as a licence plus hiring). Chinese frontier labs build almost all their environments in-house: DeepSeek, Kimi (Moonshot AI) and Zhipu (Z.ai) all report environment or tool counts in the tens of thousands in their technical reports. Third-party providers (Prime Intellect and some 35–42 others) are mostly small seed-stage teams.
  5. Synthetic data is now a structural ingredient, and the debate has moved from "will it collapse" to "is distillation allowed". Industrial practice (faithful rewriting, mixing with real data, verification and filtering) is equivalent to the "accumulate + verify" route that academic work has shown does not collapse. In Feb 2026 Anthropic alleged that about 24,000 fraudulent accounts made more than 16 million interactions for distillation (by lab: MiniMax 13M+, Moonshot 3.4M+, DeepSeek 150K+). This is Anthropic's one-sided account with no judicial finding, and the legal basis is terms of service, not copyright.
  6. Recipe openness determines how much detail this column can give. Only Ai2 (OLMo / Dolma / Tülu), Swiss AI and Hugging Face publish complete recipes. Llama 3 gives broad category shares (50/25/17/8). Qwen (Alibaba), DeepSeek and Kimi give only total tokens and stage structure. Mistral, Gemma and Llama 4 disclose essentially nothing. None of the primary documents from 23 Chinese model companies names a specific supplier such as Speechocean (Haitian Ruisheng) or Datatang, and the suppliers' annual reports are equally anonymous: the blank itself is the finding.
  7. Unlike robotics data, LLM data can be measured on a single ruler. Every layer can be converted into an implied per-token price, from $0 (open corpora) to about $31,000 per million tokens (the book settlement, assuming 100,000 tokens per book), spanning six orders of magnitude. Robotics data can only be priced by the hour, and its layers are not interchangeable.

Sourcing conventions

Figures from technical reports and papers are marked as paper figures; companies' own statements as company figures; media reports as media figures; extrapolations from public unit prices as the author's estimates; anything that could not be traced to a primary source is flagged ⚠️. Each chapter ends with a table of conflicting and unverified claims, listing circulating figures that were refuted or remain doubtful, for example "only about 40% of licences in 2026 cover training use", "Reddit–OpenAI at $70M a year", and "DCLM 7B trained on 4T tokens" (it was actually trained on 2.6T; 4T is the dataset size). These claims are not used as facts in the main text.

Chapters

#ChapterWhat it covers
01Pretraining corporaOpen pretraining corpora almost all derive from Common Crawl; the competition has moved from rule-based filtering to model scoring and synthetic rewriting.
02Copyright licensing and data dealsAI licensing prices for news, communities, books and image libraries; intermediary marketplaces and pay-per-crawl; the Bartz settlement as a price anchor.
03Human post-training dataSFT demonstrations, preference comparisons, expert question banks and rubrics: from crowdsourced piecework to a $50–250/h expert market.
04RL environments and verifiable rewardsWith post-training shifting to RL, the supply of verifiable tasks limits agent capability: how environments are built, who sells them and at what price.
05Synthetic data and distillationTextbook synthesis, web rewriting, instruction synthesis, reasoning distillation and AI feedback: who uses them, in what share, at what cost.
06Deployment feedback and user dataHow conversations, feedback and agent trajectories flow back into training: each product's training policy, data-for-discount schemes and compliance boundaries.
07Open datasets, formats and toolingHosting and formats, processing toolchains, tokenizers, open post-training sets, decontamination and licence grey zones.
08Vendors, litigation and policyMarket structure of data vendors, funding and M&A, the gradient of copyright cases, national policies and crawler governance.
09Prices, costs and mixturesThe price band from $0 open corpora to $300K environments, and how much open models disclose about their data mixtures.
10Data recipes and suppliers by model companyHow about 50 model companies in China, the US and elsewhere mix the six layers, whom they buy from, and who supplies whom.

The column also maintains 164 entity cards: 83 companies and institutions, 27 datasets, 7 tools and formats, 34 concepts and methods, and 13 papers and benchmarks. These deep pages have not been migrated yet.

Disclaimer

This column is a personal research compilation. Every figure is labelled in the text with its source and basis; companies' self-reported data is marked as a company figure, and anything not independently verified is flagged ⚠️. All judgements are as of the date of writing and are not investment or R&D advice.

Adapted from the Spark Science research notes (originally published in Chinese at jimmyresearch.com). Last updated Oct 15, 2026.