---
title: DataComp-LM Classifier
url: https://www.emergentmind.com/topics/datacomp-lm-classifier
type: topic
---

# DataComp-LM Classifier

The DataComp-LM classifier is a modular algorithmic framework for enhancing classical supervised learning estimators on binary classification tasks by integrating predictions from pre-trained, frozen large language models (LLMs). DataComp-LM functions as an ensemble and calibration layer, systematically combining classical machine learning (ML) estimators with LLM inference to provide robust improvements under both in-distribution and distribution-shifted regimes. The architecture never fine-tunes the language model, instead querying it as an oracle for scalar scores that function as soft labels or pseudo-labels [2405.05445].

## 1. System Architecture

Each input is a text pair—typically (query, product)—which is mapped to a $d$-dimensional feature vector $x \in \mathbb{R}^d$ via embedding. The system operates in three stages:
- **LLM Oracle:** For each input, a prompt is constructed and passed to a frozen pre-trained LLM, e.g., GPT-3.5-Turbo-Instruct, yielding a scalar score $z \in [0,1]$ interpreted as the LLM’s probability for the positive class.
- **Base ML Estimator:** A classical ML estimator $f_n: \mathbb{R}^d \rightarrow [0,1]$ (e.g., logistic regression) is trained on features and ground-truth labels.
- **DataComp-LM Layer:** The outputs $f_n(x)$ and $z$ are combined via one of three strategies: linear or adaptive (piecewise-constant) weighting, a calibration layer (multi-accuracy), or pseudo-label transfer for covariate-shifted test distributions.

The overall model is flexible, operating as a wrapper around arbitrary classical estimators and LLM prompts without retraining the LLM component.

## 2. Mathematical Formulation

Let $x \in \mathbb{R}^d$ denote the feature vector, $z = \phi_{LLM}(\text{raw input}) \in [0,1]$ denote the LLM score, and $f_n(x) \in [0,1]$ the base ML output. Three principal combination strategies are defined:

- **Linear Ensemble:** 
  $$
  \hat{y}^{Linear}(x) = \alpha \cdot f_n(x) + (1-\alpha) \cdot z
  $$
  The ensemble weight $\alpha \in [0,1]$ is selected via cross-validation to minimize empirical loss:
  $$
  \alpha^* = \mathop{\arg\min}_{\alpha \in [0,1]} \frac{1}{n}\sum_{i=1}^n \ell(\alpha f_n(x_i) + (1-\alpha) z_i, y_i)
  $$

- **Adaptive-Weight (AdaLinear) Ensemble:** 
  The range $[0,1]$ of $f_n(x)$ is partitioned into $r$ bins $I_1, \ldots, I_r$, and a piecewise-constant function $\alpha(\cdot)$ is learned:
  $$
  \hat{y}^{Ada}(x) = \alpha(f_n(x)) f_n(x) + [1 - \alpha(f_n(x))] z
  $$
  Each bin weight $w_j$ is optimized independently:
  $$
  w_j = \mathop{\arg\min}_{w \in [0,1]} \sum_{i: f_n(x_i) \in I_j} \ell(w f_n(x_i) + (1-w) z_i, y_i)
  $$

- **Calibration via Multi-Accuracy:**
  Both $f_n(x)$ and $z$ are discretized (to grids of size $M$, $M'$), partitioning $(x,z)$ into cells $S_{p,q}$. Two calibration approaches are described:
  - **Naive Multicalibration:** $\hat{f}_1(x) = f_n(x) + \hat{\Delta}_{p,q}$, with $\hat{\Delta}_{p,q}$ the empirical mean residual on $S_{p,q}$.
  - **Group-wise Multicalibration:** $\hat{f}_2(x) = f_n(x) + w_{i,0} + w_{j,1}$ for grid indices $i,j$, with weights fitted via least squares:
    $$
    \min \sum_{i=1}^n [f_n(x_i) + w_{p(i),0} + w_{q(i),1} - y_i]^2
    $$

- **Transfer Learning under Covariate Shift:**
  When $P_1$ (training) and $P_2$ (target) distributions diverge, auxiliary samples $x_{n+1},\dotsc,x_{n+m} \sim P_3 \approx P_2$ are drawn, and LLM pseudo-labels $z$ are collected. The objective is to retrain $f$ to jointly minimize losses on original and synthetic (LLM-labeled) data:
  $$
  \hat{f}_{n,m} = \mathop{\arg\min}_{f \in \mathcal{F}} \frac{1}{n+m} \left[ \sum_{i=1}^n \ell(f(x_i), y_i) + \sum_{i=n+1}^{n+m} \ell_0(f(x_i), z_i) \right]
  $$
  Here, $\ell_0$ is a weak-supervision loss to discount potential LLM label noise.

## 3. Training Workflow

The standardized procedure for fitting DataComp-LM includes:
1. **Feature Embedding:** Precompute embeddings $x_i$ for all data, collecting corresponding ground-truth labels $y_i$.
2. **LLM Querying:** For each input, issue prompt to the frozen LLM and record $z_i$.
3. **Base Model Training:** Fit $f_n$ to $\{(x_i, y_i)\}$.
4. **DataComp-LM Layer Fit:** Select and fit the combination/calibration strategy:
    - For ensembles, weights ($\alpha$, $\{w_j\}$) are fit by empirical minimization.
    - For calibration, conditional residuals or group-wise multitask weights are computed.
    - For transfer, augment with $m$ LLM-labeled samples and retrain using mixed-supervision.
5. **No LLM Fine-tuning:** The LLM remains frozen throughout; parameter optimization is over the combination/calibration layer only.

## 4. Inference Process

At inference, fresh inputs (query–product pairs) are processed by:
1. **Feature Extraction:** Apply embedding to obtain $x$.
2. **LLM Scoring:** Prompt the frozen LLM and collect $z$.
3. **Base Prediction:** Compute $f_n(x)$.
4. **Combination/Calibration:** Use the trained DataComp-LM rule to combine $f_n(x)$ and $z$, yielding final $\hat{y}$.
5. **Decision Rule:** Apply a fixed threshold at $0.5$ to $\hat{y}$ for binary outputs.

This procedure implements robust decision-making by leveraging two parallel, independently trained sources of prediction.

## 5. Addressing Distributional Shift

DataComp-LM addresses covariate shift by augmenting training with pseudo-labeled data sampled from the shifted distribution. As relabeling all samples from $P_2$ is assumed infeasible, the method employs the LLM as a pseudo-labeler to generate $z$ for additional data drawn from $P_3 \approx P_2$. The training objective mixes the original supervised loss and a relaxed loss for LLM-labeled data, which intuitively balances empirical risk with a regularizing effect from the pseudo-labels. Empirically, $P_3$ can be taken as $P_2$, and $m$ is chosen to approximate the test covariate distribution in the reweighted training mix [2405.05445].

## 6. Implementation Details and Pseudocode

The following pseudocode summarizes the implementation:

```python
# Input: {(x_i, y_i)}_{i=1}^n, frozen LLM, method ∈ {Linear, AdaLinear, Calibrate, Transfer}
for i in range(1, n+1):
    z_i = LLM(query_i, product_i)  # Query frozen LLM

f_n = fit_ML({(x_i, y_i)})

if method == "Linear":
    α = argmin_a sum ℓ(a * f_n(x_i) + (1 - a) * z_i, y_i)
elif method == "AdaLinear":
    Partition [0,1] into r bins
    for bin_j in bins:
        w_j = argmin_w sum_{i: f_n(x_i) ∈ bin_j} ℓ(w * f_n(x_i) + (1 - w) * z_i, y_i)
elif method == "Calibrate":
    Discretize f_n(x), z into grids of size M, M'
    Solve small least squares for corrections {w_{i,0}, w_{j,1}}
elif method == "Transfer":
    Sample {x_i}_{n+1}^{n+m} ~ P3
    For these, label z_i = LLM(raw_i)
    Re-train f_{n,m} = argmin_f [ sum_{1…n} ℓ(f(x_i), y_i) + sum_{n+1…n+m} ℓ_0(f(x_i), z_i) ]

# Output: DataComp-LM model + parameters (α or {w})
```

## 7. Empirical Performance

DataComp-LM was validated on four public benchmarks: WANDS (Wayfair relevance), Yelp sentiment, Emotion classification, and Hate-speech detection. Across all tasks, DataComp-LM variants produced consistent accuracy gains over both baseline ML and standalone LLM oracles.

| Dataset   | LLM   | ML     | Linear | AdaLinear | Calibration |
|-----------|-------|--------|--------|-----------|-------------|
| WANDS     | 77.5% | 80.3%  | 83.8%  | 84.6%     | 84.0%      |
| Yelp      | 72.4% | 69.1%  | 73.9%  | 74.2%     | 74.3%      |
| Emotion   | 75.9% | 79.9%  | 80.6%  | 81.2%     | n/a        |
| Hate      | 67.2% | 71.7%  | 72.0%  | 72.4%     | 73.1%      |

In a constructed covariate shift (Table$\rightarrow$Bed relevance), ML performance dropped from $91.5\%$ (Table) to $72.1\%$ (Bed), but Transfer-LLM recovered to $\approx78.6\%$ on the Bed set with only a small loss on Table. This result demonstrates the robustness of DataComp-LM under substantial distribution shift [2405.05445]. 

A plausible implication is that DataComp-LM provides a general recipe for deploying LLM-augmented classifiers in settings where retraining or relabeling is infeasible, leveraging LLMs for soft supervision, ensembling, or calibration to achieve state-of-the-art accuracy and stability.

Source: https://www.emergentmind.com/topics/datacomp-lm-classifier