Why a separate column on video generation data
The algorithms behind video generation models converged quickly over 2024–2026: a diffusion Transformer, an in-house captioner and preference alignment form the common skeleton of almost every technical report. What separates the models is the data: where it comes from, whether it can be used legally, how heavily it is processed, and which categories were added for a particular direction.
Yet this is exactly what companies disclose least. Of 31 mainstream models, only a few open models plus Meta Movie Gen, NVIDIA Cosmos and Stability's Stable Video Diffusion (LVD) give a clear account of their data; almost all the closed frontier models say only "public and licensed data". This column pieces together the facts that can be verified: for each layer of data, who produces it, who sells it, how models use it, what it costs, and which legal line it runs into.
Seven layers and three cross-cutting themes
The layers are ordered by distance from the final model distribution and by production method. Layers 1–3 are where the raw material comes from, layers 4–6 are how it is made and processed, and layer 7 covers the specialised categories added in 2025–2026.
| Layer | Data | Main sources | Typical scale | Role in the model |
|---|---|---|---|---|
| 1 | Public web video and open datasets | YouTube scraping; one lineage: HD-VILA-100M → Panda-70M / Koala-36M → OpenVid-1M | Open datasets at tens of millions of clips; closed sets "billions", undisclosed | Main raw material for pretraining; open datasets today serve mostly as a research and evaluation base |
| 2 | Owned platforms and UGC | Kuaishou, Douyin / TikTok, YouTube, Meta, Tencent, Alibaba | Platform libraries of tens of billions of videos (not a training-set figure) | Never directly acknowledged in tech reports; the only company-confirmed case is Google using a YouTube subset to train Veo (2 / 3 / 3.1) |
| 3 | Licensed libraries and marketplaces | Shutterstock, Getty Images, Adobe Stock, Troveo AI, Protege (incl. Calliope), Visual China Group (VCG), Jiecheng (Jetsen) | Creator footage at $1–4 per minute; overall price band spans two orders of magnitude | Compliance supplement and high-quality source footage; Adobe Firefly Video and Moonvalley's Marey sell themselves as "fully licensed" |
| 4 | Synthetic, rendered and distilled data | UE5 / game engines, Objaverse renders, VLM-generated captions, few-step distillation | Only Cosmos discloses rendered video, at 4% | Used mainly on the text side (recaptioning) and for camera, motion and physics controllability labels |
| 5 | Cleaning, splitting, scoring and captioning | In-house pipelines: shot detection → filtering → dedup and balancing → in-house captioner | Cosmos: 20M hours → about 100M clips | Determines the "usable hours"; almost nobody discloses funnel retention |
| 6 | Post-training: SFT, preferences and reward models | Hand-curated SFT, pairwise preferences, multi-dimensional reward models | Tencent HunyuanVideo: about 1M SFT examples; VideoReward / VideoAlign: 182K preference triplets | A small, expensive human step; the main entry point for outside annotation vendors |
| 7 | New data categories | Joint audio-video, multi-shot, reference pairs, camera / 3D, game actions, anime | SpatialVID: 7,089 hours; Phantom-Data: about 1M pairs | Four categories are constructed algorithmically from large pools; game-action data is the only new buy-and-sell market |
Three cross-cutting chapters complete the column:
- Data recipes and suppliers by model: a heatmap of 31 Chinese and US models × seven layers, and 29 supply relationships.
- Copyright litigation and regulation: studios suing Midjourney and MiniMax, the YouTube creator lawsuits, the EU training-data summary, and China's Interim Measures and labelling rules.
- Vendors and prices: licensing unit prices, AI licensing revenue at listed companies, and funding of data companies.
Which layers each model uses
Data layers used by each video model company
Darkest = the layer is the main pretraining source or a self-built collection system; light = minor use or open-source only; grey = not used or not disclosed. Hover over a cell for details; the full figures are in the table below.
Disclosed figures by model
| Company | Model | Region | Recipe public | Disclosed figures and route |
|---|---|---|---|---|
| Kuaishou Kling | Kling 1.x–3 / Omni | China | Partial | Web/open: Koala-36M, 172K h (open dataset, paper figure). Pipeline: three-tier governance + VTSS, no totals. Only the Koala-36M and Omni reports outline the pipeline; no scale for the main model; UGC and licensing undisclosed. |
| ByteDance Seedance / Jimeng | Seedance 1.0 / 2.0 | China | Partial | Licensed: says “public and licensed libraries”, no detail (company figure). SFT: several hundred categories, three reward models, no counts. The 1.0 report details the pipeline but not its scale; the 2.0 page has no data information; Douyin UGC use is inferred. |
| ByteDance Seaweed-7B / Waver | Seaweed-7B / Waver | China | Public | About 100M clips ≈ 220K h (our conversion); millions of simulated videos; SFT about 700K (50K weighted); Waver saw 200M+ clips. ByteDance’s two most number-rich reports: over 500K hours processed per day, invalid-clip rate cut from 42% to 2.9%. |
| ShengShu Vidu | Vidu Q1–Q3 / Vidar | China | Not public | Vidar uses about 750K open-source robot episodes; licensed: a dedicated VCG zone (company figure). No training-data report; no platform of its own, so it relies on licensed libraries, open data and embodied-AI partners; most leads are local reports pending verification. |
| AIsphere PixVerse | PixVerse V5 / V6 / C1 / R | China | Not public | Whether its 150M user records are used for training is undisclosed (inferred). Mentions only an “in-house video understanding and labelling system”; the R-series real-time world model would need action-conditioned data (inferred). |
| MiniMax Hailuo | Hailuo 01 / 02 / 03 | China | Not public | Hailuo 02 trained on about 4× the data of its predecessor (company figure). No tech report; lawsuit allegations include copyrighted film and TV characters, which are not findings of fact. |
| Alibaba Tongyi Wanxiang | Wan 2.1 / 2.2 / 2.5 | China | Public | “Billions of images and videos”, images about 10× videos; hundreds of millions of rendered text images; millions each of simple- and complex-motion videos; 2.2 adds +83.2% video (company figure). “Internal copyrighted sources + publicly accessible data”; four filtering steps remove about 50%; human scoring on 100 clusters. |
| Tencent HunyuanVideo | HunyuanVideo 1 / 1.5 | China | Public | Raw >10M h → about 800M clips (1.5); v1 SFT about 1M; 1.5 continued training 1M per task; v1 video-to-audio 250K h pretraining / 80K h SFT. One of the best-disclosed Chinese open models on scale, though sources are not named (claims GDPR compliance, includes synthesized data). |
| StepFun Step-Video-T2V | Step-Video-T2V | China | Public | 2B video-text pairs; SFT candidate pool about 30M → human curation. Eight-frame quality sampling + optical flow + 120K-cluster balancing; a vertical model co-built with Zhongguang Tianze is not evidence of corpus use. |
| Sand AI MAGI-1 | MAGI-1 | China | Partial | “Tens of PB” of raw video and images. Sources and final scale undisclosed; autoregressive per-second captions; post-training volume not found. |
| Zhipu CogVideoX | CogVideoX / Qingying | China | Public | About 35M clips after filtering (≈58K h, our conversion). “Most of the videos available online” + LAION-5B / COYO; 20K labels used to train six classifiers. |
| Kunlun Tech SkyReels | SkyReels V1 / V2 | China | Public | Self-collected 280K+ films and 800K+ TV episodes, about 6.2M h (licensing status undisclosed); about 1M SFT subset. The most direct copyright exposure in the table; SkyCaptioner-V1 human-checked accuracy 76.3%. |
| HPC-AI Tech Open-Sora 2.0 | Open-Sora 1.x / 2.0 | China | Public | Three stages with 70M / 10M / 5M samples. Total cost $199.6K (about 4,160 H200 GPU-days); video sources not named. |
| Baidu MuseSteamer | MuseSteamer | China | Not public | “Cleaning of hundreds of millions of Chinese multimodal data items” (media figure). Sources, scale and SFT undisclosed; integrated Chinese audio-video. |
| OpenAI Sora / Sora 2 | Sora / Sora 2 | US | Not public | Licensed: six-year Shutterstock deal (company figure). “Public or licensed”; on Mar 24, 2026 OpenAI announced it would stop the Sora app (effective shutdown date unverified) and the Disney deal ended. |
| Google Veo | Veo 2 / 3 / 3.1 | US | Partial | Own platform: a subset of YouTube’s ~20B videos (CNBC, secondhand); joint audio-video training (model card). Model card mentions multi-level Gemini captions, dedup and synthetic captions; video count and hours undisclosed. |
| Google Genie 3 | Genie 3 | US | Not public | Real-time interactive worlds, 720p / 24 fps (secondhand). Training data undisclosed. |
| Meta Movie Gen | Movie Gen | US | Public | O(100)M videos + O(1)B images (paper figure); audio pretraining O(1)M hours. The most complete data account among US closed models: curation + LLaMa3-Video captioner + human-refined SFT; the paper does not give source composition. |
| Runway | Gen-3 / Gen-4 / Aleph | US | Not public | Nearly 4,000 YouTube channels (404 Media leaked sheet; not necessarily all ingested); custom model on the Lionsgate library (20K+ titles, secondhand). Leak and studio partnership coexist; Getty / AMC licensing not found. |
| Luma Ray | Ray2 / Ray3 | US | Not public | Training set, licensing and post-training all undisclosed. |
| Pika | Pika | US | Not public | No training data, licensing or litigation found. |
| Midjourney Video | Video V1 | US | Not public | Court filing says it needs “billions of public images”; user ratings (image side; video undisclosed). Technology licensing deal with Meta (Midjourney as licensor); sued by Disney and two other studios. |
| Adobe Firefly Video | Firefly Video | US | Partial | Average purchase price $2.62/min (Bloomberg, secondhand). Company line: licensed content, no web scraping, no customer content; video data composition undisclosed. |
| Moonvalley Marey | Marey | US | Partial | Licensed hours undisclosed. Says it is fully licensed and signs creators directly; Troveo says it supplies most of the data (Troveo’s claim). |
| NVIDIA Cosmos | Cosmos WFM | US | Public | 20M raw hours → about 100M pretraining clips; rendered synthetic 4%; about 10M clips for fine-tuning. Paper: “proprietary + public web video”; sued over alleged use of YouTube videos referenced by HD-VILA and others (no ruling). |
| Apple video model | Apple (unnamed) | US | Not public | Shutterstock licence (reported); sued over alleged use of Panda-70M; video details undisclosed. |
| Amazon Nova Reel | Nova Reel | US | Not public | Only a generic “licensed / proprietary / open source / public”; sued by YouTube creators in Apr 2026. |
| Lightricks LTX | LTX-Video / LTX-2 | Global | Partial | Shutterstock research licence (company figure). “Large amounts of public data supplemented by licensed material”; LTX-2 weights are open, the data is not. |
| Stability SVD | Stable Video Diffusion | Global | Public | LVD: 580M clips / 212 years of footage (paper figure); 250K high-quality clips for fine-tuning. LVD source described only as “a collection of long videos”; neither itemized nor released. |
| Black Forest Labs FLUX 3 | FLUX 3 | Europe | Not public | Video prediction is 95%+ of training compute (company figure, secondhand). Joint image / video / audio / action; sources and scale undisclosed. |
| Synthesia | Synthesia avatars | Global | Partial | Paid actors filmed in studio + licensed media (company figure). Actors paid per clip, provenance documents retained; share of public content undisclosed. |
Key findings
- The raw material is largely shared; the differences lie in processing and added categories. Most mainstream open video datasets trace back to the same pool of YouTube videos (the HD-VILA-100M lineage), and the pretraining bases of the four leading Chinese closed models are also very similar. The real differences are in captioner specs, filtering thresholds and the data added for a particular direction: Kling's film/TV and dialect voices, Seedance's multimodal references, Vidu's anime and embodied data, PixVerse's real-time interaction. The base is shared; each direction pays its own tax (see the chapter on data recipes by model).
- Disclosure runs backwards: the bigger the player, the less it says. Tech reports uniformly say "public + licensed / internal copyrighted sources", and no primary document admits to using the company's own platform UGC. Data sources have been exposed mainly by media investigations (the leaked Runway training-data sheet, the Cosmos scraping controversy, The Atlantic's investigation of video datasets) and by discovery in lawsuits.
- The licensing market is growing, but it has not become the main channel for training data. Shutterstock's data business reached about $200M in 2025 and then started declining year on year; at Visual China Group, value-added services including AI training data are only 5.9% of revenue; Chinese stock-library companies generally do not report AI corpus revenue separately. Licensing serves more as a compliance supplement, a litigation settlement tool and a source of high-bitrate footage than as the bulk of pretraining.
- Synthetic data is used for labels, not pixels. Almost every model uses a VLM to generate synthetic captions. The value of rendered video lies in controllability labels such as camera pose, motion and physics; training purely on synthetic video hurts dynamics (Hunyuan-GameCraft ablation). Training on competitors' outputs is prohibited by terms of service, but no public enforcement case was found.
- Among the new categories, only game-action data has formed a seller's market. Data that aligns frames with controller inputs does not exist on the web, and in 2026 specialised suppliers such as General Intuition, Worldmodeldata and Origin Lab appeared. Audio-video, multi-shot and reference-pair data are all constructed algorithmically from large pools.
- The legal lines are still unsettled, but traceable provenance has become a cost. As of Oct 2026 there has not been a single merits ruling in video generation. Rights holders have shifted from trying to stop training to trading for output-side governance and licences: the MPA–ByteDance MOU, and the Disney × OpenAI deal, which ended when OpenAI announced in Mar 2026 that it would stop Sora. The EU training-data summary, China's "lawful source" principle and the YouTube creator lawsuits (DMCA §1201) together raise the risk of data without a chain of title.
How to read the column
- To find out what data a particular model used, go straight to the heatmap and recipe table in Data recipes and suppliers by model, then to the model card.
- To find out who sells and at what price, see Licensed libraries and marketplaces and Vendors and prices.
- Each chapter ends with a table of conflicting and unverified figures. Companies' own numbers are marked as company figures; anything not independently verified is flagged ⚠️. All litigation and deal statuses are as of Oct 2026.
Chapters
| # | Chapter | What it covers |
|---|---|---|
| 01 | Public web video and open datasets | Lineage, licences and litigation risk of open video datasets: most sources converge on HD-VILA and YouTube. |
| 02 | Owned platforms and UGC data | Who owns video corpora, what user terms say, and whether tech reports admit using platform data. |
| 03 | Licensed libraries and marketplaces | The market for licensing video to AI trainers: price bands, listed-company revenue, contract forms and China–US differences. |
| 04 | Synthetic, rendered and distilled data | Synthetic data is used mainly for text and supervision signals; rendered video is a tiny share and mostly undisclosed. |
| 05 | Cleaning, splitting, scoring and captioning pipelines | The standard pipeline from raw video to training clips, funnel retention and caption specs. |
| 06 | Post-training: SFT, preference annotation and reward models | How fine-tuning sets, preference pairs, reward models and eval data are produced, and where annotation vendors really fit in. |
| 07 | New data categories: audio-video, multi-shot, reference pairs and interactive world models | Six new kinds of training data beyond text-video pairs: how they are made, how big they are, who sells them. |
| 08 | Data recipes and suppliers by model | What 31 Chinese and US video models disclose about their data, and who supplies whom. |
| 09 | Copyright litigation and regulation | Lawsuits over video generation training data, the reach of precedents and the rules in each jurisdiction, and their effect on procurement. |
| 10 | Vendors and prices | Cross-layer vendor comparison, listed-company licensing revenue, funding events and a normalised video data price table. |
The column also maintains 134 entity cards: 36 companies and institutions, 21 datasets, 11 tools and formats, 24 concepts and methods, 8 papers and benchmarks, and 34 models. These deep pages have not been migrated yet.
This column is a personal research compilation. Every figure is labelled in the text with its source and basis; companies' self-reported data is marked as a company figure, and anything not independently verified is flagged ⚠️. All judgements are as of the date of writing and are not investment or R&D advice.
Adapted from the Spark Science research notes (originally published in Chinese at jimmyresearch.com). Last updated Oct 15, 2026.