Choosing a Pretraining Corpus: A Map of Chinese and English Open Datasets

Four layers between a download and the tokens a model reads#

The source separates four things that are easy to conflate:

• Source — where the text originates: web pages, encyclopedias, code repositories, or documents the project is licensed to use.
• Corpus — what survives extraction: the readable text pool with markup removed and obvious junk dropped.
• Recipe — how much of each kind of text is used: language shares, domain shares, and whether a small source is repeated.
• Training input — what the run actually read, after tokenization, shuffling, chunking, and packing. Tail blocks may be discarded and some content may be read more than once, so realized proportions need not match the configuration.

The chain runs from source to extraction, cleaning, and deduplication; then to an optional text pool; then to per-source training shares; then to tokenization and packing; and finally to what the model actually reads. The last step is the one that is usually invisible. A source may hold 100 GB of Chinese text while one run consumes only 20 GB of it, three times over. The questions “what is available” and “how will it be used” therefore have to be answered together, not in sequence.

The six articles of this volume each close one of those decisions, and the choice made in one changes the inputs of the next:

Six construction routes and what each one fixes#

Most open corpora fall into a small number of construction routes. The route tells you what problem the project set out to solve, not how well it solved it, and a single project may span several routes.

The English releases quoted in the source can be read against those routes. Common Crawl is the crawl itself and ships WARC (raw fetch), WAT (metadata), and WET (extracted text); taking WET saves a parsing step but is not the same as a cleaned, deduplicated training set. C4 remains the classic case study for rule-based cleaning, and its English, unfiltered, and multilingual mC4 configurations must not be treated as one dataset. RefinedWeb’s public extraction reports roughly 968M web pages and 500B to 650B tokens, with the token figure varying by tokenizer; the phrase “public extraction” matters, because it is not the full pool described in the paper. FineWeb launched at about 15T tokens, and the data card checked by the source records more than 18.5T; for anyone comparing against an older paper’s experiment, the scale of the version that experiment used is the one to use, not the later number.

Multi-source releases are described the same way. The Pile used roughly 800 GB across 22 components, which is a good study of source selection and weighting, though the historical component list does not guarantee that every part is still downloadable from its original address. Dolma describes about 3T tokens of English data spanning web pages, scientific literature, code, public-domain books, social media, and encyclopedias, published together with the processing tools — which is what makes it useful for the question “could someone else rebuild this processing step”. RedPajama-V2 exposes quality signals rather than only final text, and its official table estimates 30.4T tokens for the annotated and deduplicated head_middle slice across five European languages; that is one slice, not a project total. SlimPajama is cleaned and cross-source deduplicated from roughly 1.21T tokens of first-generation RedPajama material, published at 627B tokens; the similar name does not make it a shrunken RedPajama-V2.

DCLM is not only a dataset name but an experimental framework for comparing data-processing methods, so it has to be recorded carefully: raw pool, baseline corpus, or evaluation framework. The DCLM repository recorded a correction in September 2025 to the baseline values behind the CORE and EXTENDED aggregate scores, so older and newer aggregate numbers cannot be subtracted directly. FineWeb-Edu filters FineWeb for educational value and reported about 1.3T tokens at release; it is derived from FineWeb, so adding the two sizes does not add two independent corpora. Nemotron-CC reported about 6.3T tokens, of which roughly 1.9T was synthetic, and its v2 recipe is a separate release — generated expression is not automatically new knowledge. FinePDFs points at PDF data, where reading order, tables, and OCR errors make it a different input type from web text.

Chinese corpora: units, overlap, and near-identical names#

Chinese technical articles routinely mix English terms and code, so a rule demanding that most characters be Han will delete exactly the CUDA tutorials worth keeping, and Chinese has no word spaces, so English “too few words” and “unusual average word length” heuristics do not transfer unchanged. Beyond language adaptation, overlap between sources has to be checked.

These figures cannot be ranked against each other directly. GB and TB describe storage on disk, tokens describe how many pieces a particular tokenizer produced, and record counts describe how many entries exist. A thousand records can be a thousand short sentences or a thousand long books.

Two source-level traps are worth stating directly. First, the ChineseWebText 2.0 source description includes upstream projects such as SkyPile, WanJuan, and WuDao, so concatenating all of them does not give several independent corpora; “may overlap” has to be turned into a measured overlap on the versions actually used. Second, similar names do not mean similar training use: OpenCSG’s V2 and Hugging Face’s FineWeb2 are different projects, and OpenCSG’s Fineweb-Edu-Chinese-V3 is positioned as educational SFT data rather than a drop-in replacement for V2 web text. For historical projects such as WuDao, “was published at some point” and “which files were actually obtained this time” are separate facts, and only the second one determines reproducibility.

Multilingual releases such as mC4, OSCAR, and FineWeb2 are valuable for comparing processing across languages: FineWeb2 offers configurations by language and script, which is useful, but a 0.8 score for Chinese text and a 0.8 score for English text do not mean the same reliability unless the scores are calibrated across languages. Download a specific language configuration and version, and do not report a multilingual pool’s total as a Chinese figure.

A selection procedure that yields a record rather than a ranking#

The source answers selection with three questions rather than a leaderboard.

1. Which step do you intend to change? If the research question is body-text extraction, you need original pages or locatable crawl records. A file with only a text field is fine for studying filtering, but it cannot recover tables that upstream already deleted.
2. What information do you need to keep? If the question is about quality thresholds, prefer material that carries scores, domain labels, or a scorer that can be re-run. Taking only the top-scoring subset hides what a threshold deleted.
3. How much can you actually process? A personal project can start with one Chinese slice and one English slice, learn the fields and provenance, and only then extend. The criterion is whether the processing can be observed and rebuilt, not a dataset ranking.

Each candidate gets a short record covering which copy was used, what it contains, what processing it has already received, what is still missing, and why it is being considered at all:

Code licenses and data usage terms can attach to different objects, so the specific conditions and their versions get recorded rather than summarized as “open source”.

Verification starts small: take a sample from candidates A and B spanning different sources and lengths, check body completeness, Chinese technical content, mixed Chinese and English layout, and duplication with one common rubric, and state a prediction in advance — for instance, which candidate should retain more complete code blocks. That round supports “this material suits further processing”. Claiming the data is better for a model requires a further comparison with the model, tokenizer, effective training tokens, optimization settings, and evaluation all fixed, and two models on two leaderboards cannot decide it.

The source closes with a table of what may be claimed on this evidence and what may not, and the distinction is worth keeping:

Where the Chinese original lives#

The Chinese original is Where Pretraining Data Comes From: A Map of Chinese and English Open Corpora. It holds the per-dataset tables with every linked data card, the selection record template, the six-articled plan for the rest of the volume, and the reference list. Its sizes are the ones checked on 2026-10-15, and the selection question it leaves open is the one it answers next: which body text to keep from a web page.