From Web Page to Training Text: What Corpus Cleaning Actually Does

Four operations that are usually collapsed into one#

Cleaning a page is not one decision but four. The source keeps them apart because each one fails in a different direction.

The worked example is a self-written page containing a navigation bar, an article with an English technical sentence, a code block, an aside with a coupon prompt, and a footer with an address. Stripping the tags leaves the navigation and the advertisement in place. Collapsing all whitespace to single spaces destroys the code’s nesting. Requiring most characters on each line to be Han may delete the second technical paragraph. None of these are “insufficient cleaning”; they are different operations interfering with each other.

The practical consequence is bookkeeping: if every step records its input and its output, the point where the body text broke is identifiable. A single monolithic clean(text) that returns an empty string leaves no way to attribute the loss to a rule.

One text for training, another for matching#

Deduplication needs to recognize articles that differ superficially, which pushes toward lowercasing, stripping digits, or converting between traditional and simplified characters. Those same transforms can destroy meaning for training data: API and api may be different variables, 0.001 and 0.01 become indistinguishable without digits, and traditional-to-simplified conversion can change regional expression. Text that is convenient to match is not automatically suitable for learning.

The source’s answer is to keep two fields with different rules: training_text, the body actually used for training, receiving only reviewed transformations; and matching_text, a dedicated copy used solely for duplicate matching under its own stated rules. A record of the original position, or a controlled raw cache, allows lookups later. Unicode normalization must be stated precisely as well — NFC and NFKC are not equivalent, since NFKC folds more compatibility characters, and calling both “normalizing the encoding” hides the difference. Each rule set needs a version, otherwise a changed fingerprint in the next run cannot be attributed to a changed document rather than a changed cleaning step.

Every rule needs a unit, and every threshold needs an audit#

“How much repetition is too much” becomes decidable only once the unit is fixed. In the source’s example, a document has ten lines, five of which repeat the same navigation sentence: counting extra repeated lines gives

Counting characters instead gives a different answer for the same document. With five navigation lines of two characters each and five body lines of one hundred characters each, the repeated characters are $4\times2=8$ out of $5\times2+5\times100=510$, or about 1.6 percent. Neither computation is wrong; they measure different things, which is why a threshold is meaningless without its unit. Empty documents need their own handling rather than a denominator patched by adding one.

The same care applies to other signals, each of which can also destroy content that matters:

Word-count heuristics for English rely on spaces; splitting a Chinese paragraph on whitespace treats the whole paragraph as one word, so whether a statistic counts characters, tokens, or model tokens must be stated.

The second half of the audit asks what was removed, not only how much. Starting from one million records, extraction keeps 800,000, language filtering keeps 600,000, and rule filtering keeps 480,000. The last step retains 80 percent of the previous stage, and the pipeline retains 48 percent of the original. Those answer different questions, and a report that states only “80 percent retained” invites the reading that most of the original data survived. The breakdown that matters is by source, language, domain, and length: an overall 80 percent can hide a case where ordinary articles survive and technical documents are nearly all deleted, which for a Chinese technical project is a rule that needs re-examination. Document counts are also not a substitute for token counts, because removing many short advertisement pages barely changes training length.

Order matters for the same reason. An advertisement page may be both too short and too repetitive, and whichever rule runs first is the one that removes it. Counting only the first rejection reason cannot be used to attribute contributions to individual rules; recording all signals that fired, then the final decision, is the record that supports that analysis.

Content filtering needs context, and comparisons need a prediction#

Detection of email addresses, names, or account information can use patterns, named-entity recognition, and context rules. A pattern finds only part of the cases, and can mistake dates, version numbers, or product codes for sensitive content. After detection, the policy still has to be chosen: redact the fragment, quarantine the whole document, or exclude the whole source. A local detection result is not a document-level quality verdict, and raw text kept for traceability need not be published in full; controlled storage is enough. Keyword filtering has the same shape — an article discussing a social issue may contain a filtered term, and discussing is not endorsing. The C4 analysis by Dodge and colleagues, published at EMNLP 2021, found that filtering affected group-related text unevenly, which is why coverage of the corpus has to be checked after cleaning and not just its tidiness.

Comparing one cleaning change against another starts with a concrete prediction, such as “the new method should leave less navigation while not increasing missing code blocks”, which is far easier to falsify than “the new method should be cleaner”. The same pages run through both methods produce four categories of samples: kept by both, rejected by both, near the threshold, and decided differently — the last being the one that explains the change. Manual review is best done blind to which method produced the output, recording body completeness, code retention, and template residue separately. Only when the data layer is clear is it worth fixing the model, tokenizer, training tokens, and evaluation for a training comparison; text that reads tidier is not automatically better for a model.

The source’s summary table states what each kind of comparison result can support:

A pipeline of this kind also needs a defined policy for reads that fail. DataTrove and its FineWeb example show how read, filter, and write tasks are separated, and the source’s instruction for a small project is to write down the result of every step, the reason for each rejection, and the error records before scaling up. A read failure belongs in the error record; letting it pass as “low quality, removed” destroys the evidence that the pipeline is broken rather than selective.

Where the Chinese original lives#

The Chinese original is From Web Page to Training Text: What Corpus Cleaning Actually Does. It contains the full teaching page, the per-page-type check table, the rule-signal table, the retention arithmetic, the comparison protocol, and the reference list, including the DataTrove and FineWeb examples for splitting read, filter, and write tasks. Its examples explain why rules misfire; whether a new rule helps on a given corpus still has to be decided by the comparison procedure it describes.