DCLM-Baseline, a 7B model, scored 64% on MMLU while using 40% less compute than its rivals. The edge came from cleaner data, not a smarter architecture, per the DataComp-LM benchmark.
That 6.6-point gain over the MAP-Neo baseline traced almost entirely to one stage: the code that decided which documents entered training. Not the optimizer. Not the layer count. The filtering.
Most teams spend their planning cycles picking a base model. The bigger decision sits upstream, in the pipeline that dedups, filters, decontaminates, and mixes the corpus before a single gradient step runs.
TL;DR: data curation beats model choice
The biggest gains in an LLM pipeline come from data engineering, not model selection. DataComp-LM (June 2024) showed data-quality gains dominate scale gains above 9B tokens, and Meta's SemDeDup removed roughly 50% of redundant web data with under 0.5% performance loss and 2x faster convergence. Above 10B parameters, an aggressive curation pipeline (dedup, quality filter, decontaminate, PII scrub, mix) is where benchmark wins actually come from.
What is an LLM data pipeline?
An LLM data pipeline architecture is the staged system that turns raw text (web crawls, code, documents) into a training-ready corpus. It runs ingestion, deduplication, quality filtering, evaluation decontamination, PII removal, and domain mixing, usually in that order.
The state of the art is public. FineWeb documented 15T tokens from 96 CommonCrawl dumps. RedPajama-V2 shipped 30T raw tokens across 84 dumps with 40+ quality signals. Allen AI's Dolma ran a 12-stage pipeline over 3T tokens. These recipes aren't secret anymore. The engineering discipline to run them well still is.
Why does deduplication matter so much?
Redundant data wastes compute and pushes models to memorize instead of generalize. Removing it is the single cheapest win in the pipeline.
Meta's SemDeDup made the sharp version of this argument: "web-scale data has driven incredible progress in AI, but we don't really need all that data." Their method removes about half the semantically redundant content with under 0.5% benchmark degradation and roughly 2x faster convergence.
Dedup runs in stages, from cheap lexical matching to expensive semantic clustering.
MinHash handles near-duplicate detection at scale. You shingle each document into character k-grams, compute MinHash signatures, bucket by LSH bands, and verify Jaccard similarity on the candidates. FineWeb's production config uses 112 permutations, a 14x8 band/row layout, and k=5 character shingles. BigCode's The Stack pushed to 256 permutations for higher precision across 300+ programming languages.
from datasketch import MinHash, MinHashLSH
# FineWeb-style config: 112 permutations, 0.8 Jaccard threshold
lsh = MinHashLSH(threshold=0.8, num_perm=112)
def signature(text, k=5, num_perm=112):
m = MinHash(num_perm=num_perm)
for i in range(len(text) - k + 1):
m.update(text[i:i+k].encode("utf-8"))
return m
SemDeDup goes further. It embeds documents, clusters the embeddings with k-means, and keeps the lowest-perplexity (or highest-quality) member of each cluster. NVIDIA's NeMo Curator ships a production implementation with a default cosine similarity threshold of 0.85.
Where teams over-dedup and lose hard examples
Aggressive dedup has a failure mode that benchmarks won't warn you about. A June 2026 writeup titled "The Near-Duplicate Filter That Took Your Only Hard Example With It" captures it: a rare, valuable document that happens to resemble a common one gets deleted with the common one.
The fix is procedural. Start at a conservative threshold (0.95), lower it only if evaluation shows leftover redundancy, and keep a held-out set of known-hard examples to confirm they survive each pipeline change. Tune against downstream benchmarks, never against the dedup ratio alone.
How do quality classifiers filter training data?
Quality filtering turns raw crawl into high-signal text. Two approaches dominate, and the best pipelines run both.
FastText classifiers trained on human or model annotations score documents fast enough for trillion-token corpora. FineWeb-Edu used Llama-3-70B to label educational content, then trained a lightweight classifier to generalize at scale. It's the most-cited recipe, and also the most expensive to reproduce.
Perplexity filtering is the cheap baseline: fine-tune a small LM on clean reference text, score candidates, and drop documents above a threshold (commonly 50-100, model-dependent). High perplexity flags garbled encodings, special-character soup, and nonsense.
Dolma's 12-stage pipeline shows how these compose in production: quality classifiers, safety filtering, perplexity dedup, extraction, heuristics, document-level dedup, MinHash near-dup removal, cross-source redundancy, language ID, domain balancing, tokenization, and metadata annotation. That sequence produced OLMo 7B at MMLU 52.
One caveat worth stating plainly. Quality classifiers trained on human labels tend to favor Western, English, higher-status content, which can suppress model performance on underrepresented communities. Diversify annotation sources and measure classifier behavior across groups rather than trusting a single score.
Why decontamination protects your benchmarks
Eval contamination, where training data overlaps benchmark test sets, silently inflates every number you report. If your MMLU score includes questions the model already saw, it's fiction.
The standard defense is n-gram overlap detection. Tokenize each benchmark, generate n-grams (8 to 50 tokens), and flag any training document with more than ~30% overlap. Embedding-similarity screening catches paraphrases the n-gram pass misses.
def contaminated(train_doc, eval_ngrams, n=50, threshold=0.3):
toks = train_doc.split()
grams = {" ".join(toks[i:i+n]) for i in range(len(toks) - n + 1)}
if not grams:
return False
return len(grams & eval_ngrams) / len(grams) >= threshold
Perfect decontamination is impossible, and every serious lab says so. Subtle paraphrase and knowledge leakage slip through. Treat the number as a floor, and always keep a completely held-out test set the pipeline has never touched.
How should you scrub PII from training data?
PII scrubbing is a precision-versus-recall problem. Too aggressive and you delete useful text; too lenient and private data leaks into weights.
Layer the methods. Regex catches structured PII (emails, phones, SSNs, cards) cheaply. ML detectors catch named entities that regex can't. NVIDIA's GLiNER-PII offers NIM-based detection across 10+ entity types with a reported sub-2% false-positive rate, and Microsoft Presidio is the accessible open-source default.
| Method | Precision | Recall | Speed | Best for |
|---|---|---|---|---|
| Regex | High (known patterns) | Low | Fast | Structured fields |
| ML (GLiNER-PII) | High | High | Medium | Named entities |
| LLM-based | Very high | Very high | Slow | Ambiguous cases |
How do you optimize the data mixture?
Domain weights, how much web versus code versus books, shape what the model is good at. Hand-tuning them is guesswork.
DoReMi (Domain Reweighting with Minimax) learns the weights instead. It trains small proxy models, uses regret minimization to adjust domain weights, then scales the winning mixture. Reported result: 2-3x faster pretraining from optimized weights alone.
As a starting point before you run proxy experiments:
| Domain | Typical weight | Why |
|---|---|---|
| Web (quality-filtered) | 60-80% | Primary knowledge base |
| Code | 10-30% | Improves reasoning and syntax |
| Academic / books | 5-15% | Long-form depth |
| Conversational | 1-10% | Instruction following |
Adding 10-30% code reliably lifts coding benchmarks without degrading language performance, which is why nearly every frontier mix includes it.
Does this apply to RAG data pipelines too?
Yes, and the neglect is worse there. RAG data pipeline design reuses the same primitives: dedup so your vector store isn't 40% redundant chunks, quality-filter so garbage doesn't get retrieved, and PII-scrub before embedding. A retriever surfacing three near-identical chunks wastes the context window on repetition. The curation discipline transfers directly.
Key takeaways
- DCLM-Baseline 7B hit 64% MMLU with 40% less compute through curation alone, +6.6pp over MAP-Neo (DataComp-LM).
- SemDeDup removes ~50% of data with <0.5% loss and ~2x faster training (Meta AI, 2023).
- Above 9B tokens, data quality gains dominate scale gains. Below 1B params, basic filtering is enough.
- Over-deduplication silently deletes rare hard examples. Validate survivors; start thresholds conservative.
- Data curation is a loop: curate, train a small proxy, evaluate, adjust, repeat.
What this means for you
Match your investment to your scale. This is the decision most teams get wrong in both directions.
| Model scale | Approach | Tools |
|---|---|---|
| <1B params | Length, language ID, basic dedup | datasketch, langdetect |
| 1B-10B | Quality heuristics, moderate dedup | SlimPajama, FastText |
| 10B-70B | Full pipeline + SemDeDup + decontamination | NeMo Curator, custom |
| 70B+ | Learned mixtures, synthetic balancing, A/B tests | Internal platforms |
If you're fine-tuning under 10B parameters, don't build a bespoke platform. Start from a published corpus like SlimPajama (627B cleaned tokens), add perplexity filtering, run MinHash with FineWeb parameters, and decontaminate against your eval set. That covers most of the gain.
Treat curation as iterative. Train a 1B-7B proxy on your curated data, evaluate on held-out benchmarks, find the failure modes, adjust thresholds, and go again. DataComp-LM's core finding is that this loop compounds.
And watch the synthetic-data ratio. Iterative training on LLM-generated data risks model collapse as rare tail patterns vanish. Current evidence suggests keeping synthetic content under 50% with human data as an anchor stays safe, though the optimal ratio depends on task and scale.
The model you pick sets a ceiling. The pipeline you build decides how close you get to it.
Sources
- DataComp-LM: In search of the next generation of training sets
- SemDeDup: Data-efficient learning at web-scale through semantic deduplication
- The FineWeb Datasets: Decanting the Web
- RedPajama: an Open Dataset for Training Large Language Models
- DoReMi: Optimizing Data Mixtures Speeds Up Pretraining
- NVIDIA NeMo Curator, Semantic Deduplication
- SlimPajama: 627B cleaned, deduplicated RedPajama
- The Near-Duplicate Filter That Took Your Only Hard Example With It
- facebookresearch/SemDeDup (code)