---
title: CLIMB Aggregation for Pre-training Data Optimization
url: https://www.emergentmind.com/topics/climb-aggregation
type: topic
---

# CLIMB Aggregation for Pre-training Data Optimization

CLIMB aggregation is a methodology for constructing an optimized pre-training mixture from large, uncurated text corpora lacking explicit domain structure. CLIMB—CLustering-based Iterative Data Mixture Bootstrapping—enables automated discovery, evaluation, and refinement of data mixtures by combining unsupervised document clustering with iterative mixture optimization informed by proxy-model validation. The framework facilitates efficient exploration of the exponentially large space of possible cluster-weighted mixtures to enhance language model pre-training performance, and it obviates manual data curation and domain labeling. 

## 1. Semantic Embedding and Clustering

CLIMB begins with a corpus $\hat D = \{D_1, D_2, \dots, D_n\}$ of $n$ unlabeled documents. Each document $D_i$ is embedded into a semantic space using a fixed encoder $M_e$, yielding $E_i = M_e(D_i) \in \mathbb{R}^d$. 

A $K_{\rm init}$-means algorithm segments the embedding space into clusters by minimizing intra-cluster squared Euclidean distances:
\[
\{\mu_j\}_{j=1}^{K_{\rm init}} = \arg\min_{\mu_1,\dots,\mu_{K_{\rm init}}} \sum_{i=1}^n \min_{1\le j\le K_{\rm init}} \|E_i-\mu_j\|_2^2
\]
Each document is assigned a cluster ID:
\[
c(i) = \arg\min_{1\le j\le K_{\rm init}} \|E_i-\mu_j\|_2^2
\]
Optional post-processing includes (i) pruning low-quality clusters via a fasttext-based classifier to produce $K_{\rm pruned}$ clusters and (ii) merging closely located centroids based on an Euclidean threshold, yielding $k=K_{\rm enhanced}$ clusters $D^{(1)}, D^{(2)}, \dots, D^{(k)}$ with $\bigcup_{i=1}^k D^{(i)}\subseteq \hat D$.

## 2. Cluster-Mixture Weight Vector: Representation and Initialization

CLIMB aggregates the clusters using a mixture defined by a $k$-dimensional nonnegative vector:
\[
\alpha = (\alpha_1, \dots, \alpha_k),\quad \alpha_i \ge 0,\;\sum_{i=1}^k \alpha_i=1
\]
Each $\alpha_i$ specifies the probability of sampling tokens (or documents) from cluster $D^{(i)}$ during dataset construction.

Initialization uses a Dirichlet distribution with concentration vector proportional to cluster sizes:
\[
\alpha^{(1)} \sim {\rm Dirichlet}(\gamma_1, \dots, \gamma_k), \quad \gamma_i = \frac{|D^{(i)}|}{\sum_{j=1}^k |D^{(j)}|}
\]
This initialization favors cluster distributions echoing corpus composition but provides flexibility for mixture exploration.

## 3. Bi-level Optimization and Predictor-Based Search Approximation

Mixture selection is approached as a bi-level optimization problem. The goal is to find $\alpha^*$ minimizing the downstream validation loss after proxy-model fine-tuning:
\[
\begin{aligned}
\alpha^*  &=  \arg\min_{\alpha\in\Delta_k}\; \ell_{\rm val}\left(\alpha,\;\omega^*(\alpha)\right) \\
\text{s.t.}\quad \omega^*(\alpha)  &= \arg\min_{\omega}\; \ell_{\rm train}\bigl(\alpha,\omega\bigr) 
\end{aligned}
\]
Here, $\ell_{\rm train}$ is the proxy-model loss after training on the mixture parameterized by $\alpha$, and $\ell_{\rm val}$ is the validation metric (e.g., zero-shot task accuracy). Full grid search over $\alpha$ is computationally infeasible, so CLIMB fits a regression-style predictor $f_\theta$:
\[
f_\theta(\alpha)\approx \ell_{\rm val}(\alpha,\omega^*(\alpha))
\]
The predictor is trained by minimizing a regularized loss over observed mixtures:
\[
\theta^* = \arg\min_\theta \sum_{s\in S} \mathcal L(f_\theta(\alpha^{(s)}), \ell_{\rm val}(\alpha^{(s)})) + \lambda_1 \|\theta\|_1 + \lambda_2\|\theta\|_2^2
\]
where $S$ is the set of explored mixtures, and $L_1$/L2 regularization prevents overfitting due to the sample set's small size.

## 4. Mixture Constraints and Regularization

The feasible set for $\alpha$ is the unit simplex $\Delta_k = \{\alpha \ge 0, \sum_i \alpha_i = 1\}$. No explicit regularizer is placed on mixture weights beyond the simplex, as the probability simplex inherently induces some sparsity (i.e., only a few clusters typically receive large weights). LightGBM with $L_1$/$L_2$ penalties is employed for predictor fitting, further controlling the predictor's complexity to match sample quantity. 

## 5. Iterative Bootstrapping Procedure

CLIMB's core is an alternating $K$-iteration loop. At iteration $k$, let $S^{(k)}$ be the evaluated mixture set and $f_k$ the current predictor. Each loop iteration comprises:

- **Configuration Sampling:** Evaluate $f_k(\alpha)$ for unexplored $\alpha \in \Delta_k \setminus S^{(k)}$; select the top $N$ mixtures by predicted performance, sample $M$ new configurations $S_M^{(k)} \subset \Top_N(f_k)$, and augment $S^{(k+1)} = S^{(k)} \cup S_M^{(k)}$.
- **Proxy-model Evaluation:** For each new mixture in $S_M^{(k)}$, perform proxy-model training to measure empirical $\ell_{\rm val}$ and update the dataset.
- **Predictor Fitting:** Fit $f_{k+1}$ to all mixtures in $S^{(k+1)}$, minimizing:
\[
f_{k+1} = \arg\min_{f \in \tilde{\mathcal F}} \sum_{s\in S^{(k+1)}} \mathcal L(f(\alpha^{(s)}), \ell_{\rm val}(\alpha^{(s)})) + \lambda_1\|\theta\|_1+\lambda_2\|\theta\|_2^2
\]
After $K$ iterations, select the optimal mixture via
\[
\alpha^* = \arg\min_{\alpha\in\Delta_k} f_K(\alpha)
\]
This mixture $\alpha^*$ is employed for target-model data assembly and pre-training.

## 6. Empirical Outcomes and Released Resources

CLIMB aggregation supports both general and domain-specific optimization scenarios. When continuously trained on a 400B-token mixture selected by CLIMB, a 1B-parameter language model surpassed the Llama-3.2-1B baseline by 2.0%. When tailored for the Social Sciences domain, a 5% improvement over random sampling was observed. 

CLIMB introduces publicly available resources to facilitate reproducibility:
- **ClimbLab:** A 1.2-trillion-token corpus with 20 clusters, enabling further mixture exploration.
- **ClimbMix:** A curated 400-billion-token dataset, specifically optimized via CLIMB for efficiency and performance under constrained token budgets.

## 7. Formalization: Key Equations

The following table summarizes key mathematical constructs in CLIMB:

| Step                         | Mathematical Formulation                                                                         | Role                  |
|------------------------------|--------------------------------------------------------------------------------------------------|-----------------------|
| Embedding + Clustering       | $E_i = M_e(D_i)$; $K$-means on $E_i$                                                             | Semantic decomposition|
| Weight Vector Initialization | $\alpha^{(1)} \sim {\rm Dirichlet}(\gamma_1,\dots,\gamma_k)$                                     | Mixture start state   |
| Bi-level Obj.                | $\alpha^* = \arg\min_{\alpha\in\Delta_k}\ell_{\rm val}(\alpha,\omega^*(\alpha))$                | Selection criterion   |
| Predictor Fitting            | $f_\theta(\alpha) \approx \ell_{\rm val}(\alpha, \omega^*(\alpha))$                             | Search acceleration   |
| Bootstrapping Iteration      | Explore $\Top_N(f_k)$; refit $f_{k+1}$ on $S^{(k+1)}$                                           | Exploration–update    |
| Final selection              | $\alpha^* = \arg\min_{\alpha \in \Delta_k} f_K(\alpha)$                                         | Aggregation output    |

By integrating unsupervised clustering, proxy-model validation, and search-space pruning via learned predictors, CLIMB aggregation efficiently discovers highly nonuniform mixtures yielding improved pre-training performance over naïve or random sampling mixtures [2504.13161].

Source: https://www.emergentmind.com/topics/climb-aggregation