CLIMB Aggregation for Pre-training Data Optimization
- CLIMB Aggregation is a method for optimizing pre-training data mixtures by combining unsupervised clustering with iterative proxy-model validation on large, uncurated corpora.
- The approach employs a bi-level optimization framework with Dirichlet initialization and LightGBM-based predictors to efficiently explore and refine cluster-weighted mixtures.
- Empirical outcomes indicate a 2.0% improvement over baseline language models and a 5% gain in domain-specific scenarios, validating the method's effectiveness.
CLIMB aggregation is a methodology for constructing an optimized pre-training mixture from large, uncurated text corpora lacking explicit domain structure. CLIMB—CLustering-based Iterative Data Mixture Bootstrapping—enables automated discovery, evaluation, and refinement of data mixtures by combining unsupervised document clustering with iterative mixture optimization informed by proxy-model validation. The framework facilitates efficient exploration of the exponentially large space of possible cluster-weighted mixtures to enhance LLM pre-training performance, and it obviates manual data curation and domain labeling.
1. Semantic Embedding and Clustering
CLIMB begins with a corpus of unlabeled documents. Each document is embedded into a semantic space using a fixed encoder , yielding .
A -means algorithm segments the embedding space into clusters by minimizing intra-cluster squared Euclidean distances: Each document is assigned a cluster ID: Optional post-processing includes (i) pruning low-quality clusters via a fasttext-based classifier to produce clusters and (ii) merging closely located centroids based on an Euclidean threshold, yielding clusters 0 with 1.
2. Cluster-Mixture Weight Vector: Representation and Initialization
CLIMB aggregates the clusters using a mixture defined by a 2-dimensional nonnegative vector: 3 Each 4 specifies the probability of sampling tokens (or documents) from cluster 5 during dataset construction.
Initialization uses a Dirichlet distribution with concentration vector proportional to cluster sizes: 6 This initialization favors cluster distributions echoing corpus composition but provides flexibility for mixture exploration.
3. Bi-level Optimization and Predictor-Based Search Approximation
Mixture selection is approached as a bi-level optimization problem. The goal is to find 7 minimizing the downstream validation loss after proxy-model fine-tuning: 8 Here, 9 is the proxy-model loss after training on the mixture parameterized by 0, and 1 is the validation metric (e.g., zero-shot task accuracy). Full grid search over 2 is computationally infeasible, so CLIMB fits a regression-style predictor 3: 4 The predictor is trained by minimizing a regularized loss over observed mixtures: 5 where 6 is the set of explored mixtures, and 7/L2 regularization prevents overfitting due to the sample set's small size.
4. Mixture Constraints and Regularization
The feasible set for 8 is the unit simplex 9. No explicit regularizer is placed on mixture weights beyond the simplex, as the probability simplex inherently induces some sparsity (i.e., only a few clusters typically receive large weights). LightGBM with 0/1 penalties is employed for predictor fitting, further controlling the predictor's complexity to match sample quantity.
5. Iterative Bootstrapping Procedure
CLIMB's core is an alternating 2-iteration loop. At iteration 3, let 4 be the evaluated mixture set and 5 the current predictor. Each loop iteration comprises:
- Configuration Sampling: Evaluate 6 for unexplored 7; select the top 8 mixtures by predicted performance, sample 9 new configurations 0, and augment 1.
- Proxy-model Evaluation: For each new mixture in 2, perform proxy-model training to measure empirical 3 and update the dataset.
- Predictor Fitting: Fit 4 to all mixtures in 5, minimizing: 6 After 7 iterations, select the optimal mixture via
8
This mixture 9 is employed for target-model data assembly and pre-training.
6. Empirical Outcomes and Released Resources
CLIMB aggregation supports both general and domain-specific optimization scenarios. When continuously trained on a 400B-token mixture selected by CLIMB, a 1B-parameter LLM surpassed the Llama-3.2-1B baseline by 2.0%. When tailored for the Social Sciences domain, a 5% improvement over random sampling was observed.
CLIMB introduces publicly available resources to facilitate reproducibility:
- ClimbLab: A 1.2-trillion-token corpus with 20 clusters, enabling further mixture exploration.
- ClimbMix: A curated 400-billion-token dataset, specifically optimized via CLIMB for efficiency and performance under constrained token budgets.
7. Formalization: Key Equations
The following table summarizes key mathematical constructs in CLIMB:
| Step | Mathematical Formulation | Role |
|---|---|---|
| Embedding + Clustering | 0; 1-means on 2 | Semantic decomposition |
| Weight Vector Initialization | 3 | Mixture start state |
| Bi-level Obj. | 4 | Selection criterion |
| Predictor Fitting | 5 | Search acceleration |
| Bootstrapping Iteration | Explore 6; refit 7 on 8 | Exploration–update |
| Final selection | 9 | Aggregation output |
By integrating unsupervised clustering, proxy-model validation, and search-space pruning via learned predictors, CLIMB aggregation efficiently discovers highly nonuniform mixtures yielding improved pre-training performance over naïve or random sampling mixtures (Diao et al., 17 Apr 2025).