DataComp-LM for Large Language Model Training

DataComp-LM (DCLM) is an open-source framework for building better training datasets and developing large language models (LLMs). It provides data-processing tools, standardized training recipes, and evaluation workflows that help researchers measure how dataset choices affect model performance.

Rather than treating model architecture as the only factor in AI development, DCLM emphasizes the quality and composition of training data. Its workflow lets researchers experiment with filtering, cleaning, and selecting datasets before training models at different compute scales. For related AI research, explore our latest technical tutorials and open-source tools.

What Is DataComp-LM?

DataComp-LM provides a standardized corpus containing more than 300 trillion unfiltered tokens sourced from CommonCrawl, alongside baseline pretraining recipes based on the open_lm framework. It also includes an evaluation suite covering more than 50 tasks, allowing researchers to compare models trained with different data strategies.

The framework supports experiments ranging from smaller models with roughly 400 million parameters to models with approximately 7 billion parameters. Researchers can therefore study dataset quality across different training budgets instead of relying on a single model size.

Learn more about the framework through the official DataComp-LM project website and its research resources.

DataComp-LM Workflow and Dataset Preparation

The DCLM workflow separates dataset construction from model training and evaluation. This structure helps researchers track the data used in each experiment and reproduce results more consistently.

Selecting and Processing Raw Data

Researchers begin by selecting a raw data source or choosing an existing competition dataset pool. They then process the material through filtering and cleaning steps to prepare a dataset for training. In the filtering track, participants select useful data from a shared pool. In the bring-your-own-data track, they construct a dataset using their own sources.

Deduplication, Tokenization, and Shuffling

Data preparation can include removing duplicate content and converting text into tokens using a model-compatible tokenizer. The resulting tokenized data is shuffled before training to prepare it for efficient processing.

DCLM supports both Rust-based and Ray-based tokenization and shuffling workflows. The Rust implementation is recommended for many single-machine workloads, while Ray can help process datasets that require distributed resources.

Training and Evaluating Language Models

After preparing the dataset, researchers train a model using standardized scripts and scale-specific hyperparameters. DCLM records experiment details through reference JSON files that identify datasets, tokenized artifacts, trained models, and evaluation results.

This experiment tracking makes it easier to connect a model’s performance with the dataset and configuration used to produce it. The repository also provides evaluation scripts for measuring performance across downstream tasks.

For research comparisons, use the current evaluation code and document the exact version of the metrics. The project updated its centered CORE and EXTENDED calculations in September 2025, so older scores may not be directly comparable with newer results.

Getting Started with DataComp-LM

Before running experiments, prepare a suitable Linux environment with Python 3.10, the required dependencies, and access to the compute and storage resources needed for your selected model scale. The official repository documents the setup process and provides the commands for reproducing baseline experiments.

git clone https://github.com/mlfoundations/dclm.git
cd dclm
pip install -r requirements.txt

These commands download the source and install the listed Python dependencies. Additional system packages, data access, and compute configuration may be required before training can begin.

Researchers who complete training and evaluation can prepare their results for submission to the project’s leaderboard by following its submission workflow. Consult the official DataComp-LM GitHub repository for current installation instructions, datasets, evaluation scripts, contribution guidelines, and licensing details.