Papers
Topics
Authors
Recent
Search
2000 character limit reached

CLIMB Aggregation for Pre-training Data Optimization

Updated 12 March 2026
  • CLIMB Aggregation is a method for optimizing pre-training data mixtures by combining unsupervised clustering with iterative proxy-model validation on large, uncurated corpora.
  • The approach employs a bi-level optimization framework with Dirichlet initialization and LightGBM-based predictors to efficiently explore and refine cluster-weighted mixtures.
  • Empirical outcomes indicate a 2.0% improvement over baseline language models and a 5% gain in domain-specific scenarios, validating the method's effectiveness.

CLIMB aggregation is a methodology for constructing an optimized pre-training mixture from large, uncurated text corpora lacking explicit domain structure. CLIMB—CLustering-based Iterative Data Mixture Bootstrapping—enables automated discovery, evaluation, and refinement of data mixtures by combining unsupervised document clustering with iterative mixture optimization informed by proxy-model validation. The framework facilitates efficient exploration of the exponentially large space of possible cluster-weighted mixtures to enhance LLM pre-training performance, and it obviates manual data curation and domain labeling.

1. Semantic Embedding and Clustering

CLIMB begins with a corpus D^={D1,D2,…,Dn}\hat D = \{D_1, D_2, \dots, D_n\} of nn unlabeled documents. Each document DiD_i is embedded into a semantic space using a fixed encoder MeM_e, yielding Ei=Me(Di)∈RdE_i = M_e(D_i) \in \mathbb{R}^d.

A KinitK_{\rm init}-means algorithm segments the embedding space into clusters by minimizing intra-cluster squared Euclidean distances: {μj}j=1Kinit=arg⁡min⁡μ1,…,μKinit∑i=1nmin⁡1≤j≤Kinit∥Ei−μj∥22\{\mu_j\}_{j=1}^{K_{\rm init}} = \arg\min_{\mu_1,\dots,\mu_{K_{\rm init}}} \sum_{i=1}^n \min_{1\le j\le K_{\rm init}} \|E_i-\mu_j\|_2^2 Each document is assigned a cluster ID: c(i)=arg⁡min⁡1≤j≤Kinit∥Ei−μj∥22c(i) = \arg\min_{1\le j\le K_{\rm init}} \|E_i-\mu_j\|_2^2 Optional post-processing includes (i) pruning low-quality clusters via a fasttext-based classifier to produce KprunedK_{\rm pruned} clusters and (ii) merging closely located centroids based on an Euclidean threshold, yielding k=Kenhancedk=K_{\rm enhanced} clusters nn0 with nn1.

2. Cluster-Mixture Weight Vector: Representation and Initialization

CLIMB aggregates the clusters using a mixture defined by a nn2-dimensional nonnegative vector: nn3 Each nn4 specifies the probability of sampling tokens (or documents) from cluster nn5 during dataset construction.

Initialization uses a Dirichlet distribution with concentration vector proportional to cluster sizes: nn6 This initialization favors cluster distributions echoing corpus composition but provides flexibility for mixture exploration.

3. Bi-level Optimization and Predictor-Based Search Approximation

Mixture selection is approached as a bi-level optimization problem. The goal is to find nn7 minimizing the downstream validation loss after proxy-model fine-tuning: nn8 Here, nn9 is the proxy-model loss after training on the mixture parameterized by DiD_i0, and DiD_i1 is the validation metric (e.g., zero-shot task accuracy). Full grid search over DiD_i2 is computationally infeasible, so CLIMB fits a regression-style predictor DiD_i3: DiD_i4 The predictor is trained by minimizing a regularized loss over observed mixtures: DiD_i5 where DiD_i6 is the set of explored mixtures, and DiD_i7/L2 regularization prevents overfitting due to the sample set's small size.

4. Mixture Constraints and Regularization

The feasible set for DiD_i8 is the unit simplex DiD_i9. No explicit regularizer is placed on mixture weights beyond the simplex, as the probability simplex inherently induces some sparsity (i.e., only a few clusters typically receive large weights). LightGBM with MeM_e0/MeM_e1 penalties is employed for predictor fitting, further controlling the predictor's complexity to match sample quantity.

5. Iterative Bootstrapping Procedure

CLIMB's core is an alternating MeM_e2-iteration loop. At iteration MeM_e3, let MeM_e4 be the evaluated mixture set and MeM_e5 the current predictor. Each loop iteration comprises:

  • Configuration Sampling: Evaluate MeM_e6 for unexplored MeM_e7; select the top MeM_e8 mixtures by predicted performance, sample MeM_e9 new configurations Ei=Me(Di)∈RdE_i = M_e(D_i) \in \mathbb{R}^d0, and augment Ei=Me(Di)∈RdE_i = M_e(D_i) \in \mathbb{R}^d1.
  • Proxy-model Evaluation: For each new mixture in Ei=Me(Di)∈RdE_i = M_e(D_i) \in \mathbb{R}^d2, perform proxy-model training to measure empirical Ei=Me(Di)∈RdE_i = M_e(D_i) \in \mathbb{R}^d3 and update the dataset.
  • Predictor Fitting: Fit Ei=Me(Di)∈RdE_i = M_e(D_i) \in \mathbb{R}^d4 to all mixtures in Ei=Me(Di)∈RdE_i = M_e(D_i) \in \mathbb{R}^d5, minimizing: Ei=Me(Di)∈RdE_i = M_e(D_i) \in \mathbb{R}^d6 After Ei=Me(Di)∈RdE_i = M_e(D_i) \in \mathbb{R}^d7 iterations, select the optimal mixture via

Ei=Me(Di)∈RdE_i = M_e(D_i) \in \mathbb{R}^d8

This mixture Ei=Me(Di)∈RdE_i = M_e(D_i) \in \mathbb{R}^d9 is employed for target-model data assembly and pre-training.

6. Empirical Outcomes and Released Resources

CLIMB aggregation supports both general and domain-specific optimization scenarios. When continuously trained on a 400B-token mixture selected by CLIMB, a 1B-parameter LLM surpassed the Llama-3.2-1B baseline by 2.0%. When tailored for the Social Sciences domain, a 5% improvement over random sampling was observed.

CLIMB introduces publicly available resources to facilitate reproducibility:

  • ClimbLab: A 1.2-trillion-token corpus with 20 clusters, enabling further mixture exploration.
  • ClimbMix: A curated 400-billion-token dataset, specifically optimized via CLIMB for efficiency and performance under constrained token budgets.

7. Formalization: Key Equations

The following table summarizes key mathematical constructs in CLIMB:

Step Mathematical Formulation Role
Embedding + Clustering KinitK_{\rm init}0; KinitK_{\rm init}1-means on KinitK_{\rm init}2 Semantic decomposition
Weight Vector Initialization KinitK_{\rm init}3 Mixture start state
Bi-level Obj. KinitK_{\rm init}4 Selection criterion
Predictor Fitting KinitK_{\rm init}5 Search acceleration
Bootstrapping Iteration Explore KinitK_{\rm init}6; refit KinitK_{\rm init}7 on KinitK_{\rm init}8 Exploration–update
Final selection KinitK_{\rm init}9 Aggregation output

By integrating unsupervised clustering, proxy-model validation, and search-space pruning via learned predictors, CLIMB aggregation efficiently discovers highly nonuniform mixtures yielding improved pre-training performance over naïve or random sampling mixtures (Diao et al., 17 Apr 2025).

Definition Search Book Streamline Icon: https://streamlinehq.com
References (1)

Topic to Video (Beta)

No one has generated a video about this topic yet.

Whiteboard

No one has generated a whiteboard explanation for this topic yet.

Follow Topic

Get notified by email when new papers are published related to CLIMB Aggregation.