---
title: Retrieval-Augmented ICL Scaling Law
url: https://www.emergentmind.com/topics/retrieval-augmented-in-context-learning-icl-scaling-law
type: topic
---

# Retrieval-Augmented ICL Scaling Law

Below is a self‐contained exposition of the proposed T-similarity method for self-training and its relation to Self-Training with Classifier Disagreement (SCD).  We follow the seven points requested:

1.  Self‐training loop  
   • Initialization  
     – We have a small labeled set \( (\mathbf{X}_\ell,y_\ell) \) and a large unlabeled set \(\mathbf{X}_u\).  
     – A neural network feature extractor \( f_\theta(\cdot)\) is randomly initialized (or pre‐trained).  
     – On top of the fixed feature extractor we initialize  
         • one “prediction” head \(h_{\rm pred}\) for final classification (trained by standard cross‐entropy), and  
         • an ensemble \(\mathcal T=\{h_m\}_{m=1}^M\) of \(M\) linear “confidence” heads.  
   • Ensemble construction  
     – Each linear head \(h_m\) is a weight vector \(\omega_m\in\mathbb R^d\) predicting logits \(h_m(x)=\omega_m^\top f_\theta(x)\).  
     – We fit \(\{\omega_m\}\) by minimizing a joint objective  
       \[
         \mathcal L_{\rm conf}(\{\omega_m\})
         = \frac1M\sum_{m=1}^M\frac{1}{n_\ell}\sum_{i=1}^{n_\ell}\ell_{\rm CE}\bigl(\omega_m^\top f_\theta(x_i),y_i\bigr)
           \;-\;\gamma\,\frac1{n_uM(M-1)}\sum_{m\neq k}\sum_{x\in\mathbf X_u}h_m(x)^\top h_k(x),
       \]
       where the second term maximizes prediction diversity on \(\mathbf X_u\).  
   • Pseudo‐labeling via T-similarity  
     – For each \(x\in\mathbf X_u\), compute  
       \[
         s_{\mathcal T}(x)
         = \frac{1}{M(M-1)}\sum_{m\neq k}h_m(x)^\top h_k(x)
         \;\in[0,1],
       \]
       the average pairwise cosine‐like similarity between classifiers’ output distributions.  
     – Convert \(s_{\mathcal T}(x)\) to a confidence score and compare to a threshold \(\theta\) (fixed or adaptive).  
     – If \(s_{\mathcal T}(x)\ge\theta\), assign \(\hat y = \arg\max_c\,h_{\rm pred}(x)_c\) as pseudo‐label.  
   • Incorporation and retraining  
     – Move the selected \((x,\hat y)\) into the labeled set, remove it from \(\mathbf X_u\).  
     – Retrain \(h_{\rm pred}\) on the augmented labeled set; re-optimize the ensemble \(\mathcal T\).  
     – Repeat for up to \(N\) rounds or until \(\mathbf X_u\) is exhausted.

2.  Mathematical definition of T-similarity  
   – Let \(\mathcal T=\{h_m\colon\mathcal X\to\Delta_C\}_{m=1}^M\) be an ensemble of classifiers whose outputs lie in the probability simplex \(\Delta_C\).  Define
     \[
       s_{\mathcal T}(x)
       = \frac{1}{M(M-1)}\sum_{m\neq k}h_m(x)^\top h_k(x).
     \]
   – Equivalently, if each \(h_m(x)\in\mathbb R^C\) is a probability vector,
     \[
       h_m(x)^\top h_k(x)
       = \sum_{c=1}^C h_{m,c}(x)\,h_{k,c}(x)
       \quad\implies\quad
       0\;\le\;s_{\mathcal T}(x)\;\le\;1.
     \]
     (Proposition: \(s_{\mathcal T}(x)\in[0,1]\).)  
   – Intuition: large \(s_{\mathcal T}(x)\) means low disagreement → high confidence; low \(s_{\mathcal T}(x)\) means classifiers disperse → low confidence.  
   – Pseudo‐labeling policies compare \(s_{\mathcal T}(x)\) against a threshold \(\theta\) (fixed, curriculum–adaptive, or transductive‐bound–based).

3.  High‐level pseudocode  
   ```
   Input: (X_ℓ,y_ℓ), X_u, feature net f_θ, prediction head h_pred,
          M confidence heads {ω_m}, thresholding policy ψ, max rounds N
   for t=1…N:
     1. Train h_pred on (X_ℓ,y_ℓ) by standard cross‐entropy.
     2. Fit ensemble ω₁…ω_M to (X_ℓ,y_ℓ) plus diversity on X_u:
          min  (1/M)∑_m CE(ω_m;X_ℓ,y_ℓ)  − γ·(1/[M(M−1)])∑_{m≠k}∑_{x∈X_u} (ω_m⊤f_θ(x))(ω_k⊤f_θ(x)).
     3. Compute s_T(x) for all x∈X_u.
     4. Select X_pl = ψ(X_u,{s_T(x)}).  For x∈X_pl set ŷ=argmax h_pred(f_θ(x)).
     5. Augment:  X_ℓ←X_ℓ∪X_pl,  y_ℓ←y_ℓ∪{ŷ},  remove X_pl from X_u.
   Output: final h_pred
   ```
   Differences in the three ψ‐policies:
   • PL_θ:  select all x with s_T(x)≥θ.  
   • CSTA_Δ:  at iteration t choose θ^(t) as suitable quantile of {s_T(x)} to enforce |X_pl|≈Δ·|X_u|.  
   • MSTA:  pick class‐specific thresholds by minimizing a transductive‐error bound:  
     \(\min_θ\;R_{u,≥θ}/(|pseudo|)\).

4.  Theoretical Analysis  (binary case, linear heads)  
   – Objective (Problem (P)) for \(\mathbf W=[ω_1…ω_M]∈ℝ^{d×M}\):
     \[
       \min_W
       \frac1{M\,n_ℓ}\!\sum_{m=1}^M\sum_{i=1}^{n_ℓ}(y_i−ω_m⊤x_i)^2
       +\frac1M\sum_m λ_m\|ω_m\|^2
       +\frac{γ}{n_uM(M−1)}\sum_{m≠k}\sum_{i} (ω_m⊤x_i)(ω_k⊤x_i).
     \]
   – Assumption A:  ∀m,  \(λ_m > \frac{γ(M+1)}{n_u(M-1)}·λ_{\max}(X_u⊤X_u)\).  
   – Proposition 4.3 (loss properties): under A, \(\mathcal L\) is strictly convex and coercive → unique global minimizer W*.  
   – Stationary‐point Euler eq. ∇L(W)=0 reduces to a linear system (Proposition 4.4):
     \[
       \Bigl(λ_mI_d +\frac{X_ℓ⊤X_ℓ}{n_ℓ}\Bigr)ω_m
       \;=\;
       \frac{X_ℓ⊤y_ℓ}{n_ℓ}
       \;-\;
       \frac{γ}{n_u(M-1)}\,X_u⊤X_u\sum_{k≠m}ω_k.
     \]
   – Theorem 4.5 (lower bound on diversity):
     let \(\tilde W\) be the solution, assume \(\frac1M∑_mλ_m\|ω_m\|^2≥1\).  Then
     \[
       γ\,ℓ_{\rm div}(\tilde W,X_u)
       \;\ge\;
       \frac1{2n_ℓM}\sum_{m=1}^M\|y_ℓ−X_ℓω_m\|^2
       +\frac1{2M}\sum_{m=1}^Mω_m⊤\Bigl(λ_mI+\tfrac{X_ℓ⊤X_ℓ}{n_ℓ}\Bigr)ω_m.
     \]
     In particular \(ℓ_{\rm div}≥0\), and high diversity ↔ large margins on labeled data.  
   – Corollary 4.6 (role of representation): if all \(λ_m=λ\), then
     \[
       γ\,ℓ_{\rm div}(\tilde W,X_u)\;\ge\;\frac1{2M}\Bigl(λ+\tfrac{λ_{\min}(X_ℓ⊤X_ℓ)}{n_ℓ}\Bigr)\|\tilde W\|_F^2.
     \]
     Thus, spreading labeled features evenly (large \(λ_{\min}(X_ℓ⊤X_ℓ)\)) boosts diversity.

5.  Experimental setup  
   • Datasets (13 SSL benchmarks):  
     – Biological: Cod-RNA, DNA, Protein, Splice  
     – Vision: COIL-20, Digits, MNIST  
     – Tabular: DryBean, Mushrooms, Phishing, Rice, Svmguide1  
     – Time series: HAR  
   • Labeling regimes:  
     – IID: random class‐balanced sampling.  
     – SSB: per‐class selection biased by \(\exp(r·|{\rm PC}_1(x)|)\) on first principal component (strength hyperparameter \(r\) tuned per dataset).  
   • Architecture & training:  
     – A 3-layer MLP feature extractor \(f_θ\).  
     – Prediction head \(h_{\rm pred}\) + ensemble of \(M=5\) linear heads.  
     – Optimizer: Adam \(lr=10^{-3}\), 5 epochs ×100 iters/epoch.  
     – Diversity strength \(\gamma=1\), cross-entropy on ℓ sup, LS-SVM style on ensemble.  
   • Baselines:  
     – ERM (supervised only)  
     – PL_{θ=0.8} (fixed threshold 0.8)  
     – CSTA_{Δ=0.4} (curriculum)  
     – MSTA (transductive‐bound).  
     Each with softmax‐confidence vs T-similarity.  
   • Metrics: test accuracy (%), calibration (Expected Calibration Error).

6.  Quantitative results & ablations  
   • Failure of softmax under SSB:  all four self-training schemes drop by up to 30 points vs IID; e.g. on Mushrooms softmax policies even underperform ERM.  
   • T-similarity gains under SSB:  
     – PL_{θ=0.8}: +8/13 datasets  
     – CSTA_{Δ=0.4}, MSTA: +11/13 datasets  
     – Improvements up to ~18 points in worst‐case SSB.  
   • Under IID, T-similarity performs on par with softmax (no degradation).  
   • Confidence‐distribution plots: T-similarity concentrates high confidence on correct predictions, low on errors; softmax remains overconfident for both.  
   • Calibration (ECE) vs γ: imposing diversity (γ>0) steadily improves calibration in both IID & SSB.  
   • Ablations:  
     – Pseudo‐label threshold θ∈{0.7,0.8,0.9,0.95}: T-sim robust across θ, softmax more brittle under SSB.  
     – Labeled set size n_ℓ∈[20…2000]: T-sim outperforms softmax for small n_ℓ under SSB, matches softmax as n_ℓ grows.  
     – Diversity weight γ∈{0,0.5,1,1.5,2}: any γ>0 helps under SSB; little harm under IID.  
     – Ensemble size M∈{2,5,10}: M=5 is a good compromise; gains persist for larger M but with diminishing returns.

7.  Practical recommendations & insights  
   – Plug‐and‐play: replace softmax‐confidence by T-similarity in any wrapper self-training.  
   – Modest overhead: only M=5 linear heads on top of fixed features.  
   – Choose γ>0 (even γ=0.5) to promote diversity.  
   – Ensemble size M≃5–10 suffices.  
   – Under distribution shift (SSB / covariate shift), T-similarity hugely stabilizes pseudo‐label quality.  
   – Ensure the learned feature space has spread (large λ_min of X_ℓ⊤X_ℓ) to maximize ensemble coverage (insight from Corollary 4.6)—akin to encouraging uniform embeddings.  
   – Connections to SCD: by explicitly encouraging classifier disagreement (negative‐correlation loss) on unlabeled data, T-similarity generalizes the SCD idea from two classifiers to a multi‐member ensemble, yielding a well-calibrated confidence measure for pseudo-labeling.

References to all equations and theorems are as numbered in the source paper.

Source: https://www.emergentmind.com/topics/retrieval-augmented-in-context-learning-icl-scaling-law