---
title: 'ICE: Bias-Corrected Parameter Estimation'
url: https://www.emergentmind.com/topics/information-corrected-estimator-ice
type: topic
---

# ICE: Bias-Corrected Parameter Estimation

Information-Corrected Estimator (ICE) is a parameter estimation procedure designed to reduce generalization error in supervised machine learning by explicitly correcting for the Kullback–Leibler (KL) divergence generalization bias intrinsic to standard maximum likelihood estimation (MLE) and $L_2$ (ridge) regularization. The ICE estimator augments the loss function with a data-driven trace correction, requiring no tuning or hyper-parameter search, and applies to any model with a twice-differentiable likelihood. Empirical studies and distributed implementations in modern neural network frameworks demonstrate robust out-of-sample performance improvements over unregularized MLE and ridge, especially in moderate sample regimes and complex model classes [1803.04947][2007.06157].

## 1. Formal Definition and Objective

ICE modifies the standard empirical log-likelihood objective by introducing a finite-sample correction based on Fisher information and the observed Hessian. Given IID data $\{x_i\}_{i=1}^n$ and a parametric model $g(x \mid \theta)$, the average log-likelihood is
\[
\ell(\theta) = \frac{1}{n} \sum_{i=1}^n \log g(x_i\mid\theta).
\]
Define the empirical Fisher information and observed Hessian as
\[
\hat I(\theta) = \frac{1}{n}\sum_{i=1}^n [\nabla_\theta \log g(x_i\mid\theta)][\nabla_\theta \log g(x_i\mid\theta)]^\top,
\]
\[
\hat J(\theta) = -\frac{1}{n} \sum_{i=1}^n \nabla^2_\theta \log g(x_i\mid\theta).
\]
The ICE objective is
\[
-\ell^*(\theta) = -\ell(\theta) + \frac{1}{n}\mathrm{tr}\bigl(\hat I(\theta)\,\hat J(\theta)^{-1}\bigr),
\]
with the ICE estimator
\[
\hat\theta_{\rm ICE} = \arg\min_\theta \{-\ell^*(\theta)\}.
\]
No additional penalty or tuning parameter is introduced.

## 2. Theoretical Motivation: KL Divergence and Bias Correction

ICE is derived to target the out-of-sample KL divergence from the true data law $f(x)$ to $g(x \mid \theta)$:
\[
\rho_{KL}(f,g_\theta) = \int f(x) \log \frac{f(x)}{g(x\mid\theta)}dx = C(f) - E_f[\log g(X\mid\theta)].
\]
MLE underestimates out-of-sample error due to an optimistic $O(n^{-1})$ bias in the empirical log-likelihood. Under regularity and smoothness conditions (including White’s conditions), a second-order expansion shows that
\[
b(\hat\theta) \approx \frac{1}{n} \mathrm{tr}\bigl(I(\hat\theta)\,J(\hat\theta)^{-1}\bigr) + O(n^{-3/2}),
\]
where $I$ and $J$ are the population Fisher information and Hessian. By adding the empirical counterpart $\mathrm{tr}(\hat I\,\hat J^{-1})/n$ to the objective, ICE cancels the leading bias term, reducing the generalization error’s order from $O(1/n)$ to $O(n^{-3/2})$ [1803.04947].

## 3. Theoretical Guarantees

The central properties of ICE are as follows:

- **Asymptotic Normality**: Under standard regularity conditions and for the population minimizer $\theta_0$,
  \[
  \sqrt{n}(\hat\theta_{\rm ICE} - \theta_0) \xrightarrow{d} \mathcal N(0, J^*(\theta_0)^{-1} I^*(\theta_0) J^*(\theta_0)^{-1}),
  \]
  where $I^*, J^*$ are the “corrected” information and Hessian tensors associated with $\ell^*$.

- **Bias Reduction**: The expectation of the ICE-corrected loss at the minimizer satisfies
  \[
  \mathcal L^*(\hat\theta_{\rm ICE}) = \ell^*(\hat\theta_{\rm ICE}) + O_p(n^{-3/2}),
  \]
  while ordinary MLE has an $O(n^{-1})$ discrepancy. Thus, ICE achieves a strictly smaller asymptotic bias at finite $n$ [1803.04947].

## 4. Algorithmic Implementation and Approximations

Optimization proceeds by iterative minimization of $-\ell^*(\theta)$. At each iteration:

- Compute per-sample gradients and Hessians (or their diagonals).
- Form empirical estimators $\hat I$ and $\hat J$.
- Compute the correction term $c(\theta) = \mathrm{tr}(\hat I\,\hat J^{-1})/n$.
- Update parameters via a quasi-Newton or similar gradient-based solver.

Complexity per iteration depends on the required matrix operations:

| Step                              | Time Complexity         | Memory      |
|------------------------------------|------------------------|-------------|
| Forming $\hat I$, $\hat J$         | $O(n p^2)$             | $O(p^2)$    |
| Inverting $\hat J$                 | $O(p^3)$               | $O(p^2)$    |
| Diagonal approximation to $\hat J$ | $O(n p)$               | $O(p)$      |

Practical implementations, such as in multilayer perceptron (MLP) models, use a diagonal approximation $\hat J \approx \hat D = \mathrm{diag}(\hat J)$ to obtain
\[
-\ell^*(\theta) \approx -\ell(\theta) + \frac{1}{n}\sum_{i=1}^n v(\theta, x_i)^T \hat D^{-1} v(\theta, x_i),
\]
which matches the computational cost of vanilla backpropagation up to a small multiplicative factor [2007.06157].

Numerical stabilization of $\hat D^{-1}$ is required to avoid division by near-zero or negative diagonal entries; this is accomplished by truncating small scores and reweighting as detailed in [2007.06157].

## 5. Empirical Results and Applications

ICE has been empirically validated in both classical statistical models and large-scale neural networks:

**Parametric models [1803.04947]:**
- **Gaussian location-scale**: For sample sizes $n \in \{16,32,...,1024\}$, ICE reduces out-of-sample KL divergence by over $4.5 \sigma$ at smallest $n$, remaining $\sim1\sigma$ ahead of MLE at $n=1024$. $L_2$ regularization was neutral or harmful.
- **Friedman nonlinear regression**: ICE reduced KL divergence by $8$–$14\%$ for all $n$, while $L_2$ was only marginally effective for $n < 32$.
- **Synthetic logistic regression**: For $p \in \{5,10,20\}$ and $n \leq 5000$, ICE reduced generalization KL by $10$–$20\%$ when $n$ was small; this advantage attenuated as $n$ increased.

**MLPs and distributed training [2007.06157]:**
- Implemented in Apache Spark’s `MultilayerPerceptronClassifier` via a boolean switch `useICE`.
- On the Freddie Mac Single‐Family Loan‐Level dataset ($\sim2\times10^6$ samples), across a range of architecture depths (36, 78, 159, 291 parameters), ICE significantly reduced the train/test generalization gap for $n\leq4096$, with test cross-entropy loss improvements statistically significant ($p<0.01$). For large $n$, both ICE and MLE converged to similar test error, but ICE consistently had less variability.
- ICE added $10$–$20\%$ to the loss/gradient computation per iteration, but required fewer optimization steps and thus similar or lower total runtime.

## 6. Comparison with MLE and Ridge Regularization

The distinguishing characteristics between ICE, MLE, and $L_2$ regularization are outlined as follows:

| Method        | Objective                   | Bias Order | Hyper-parameters | Applicability                  | Overfitting control             |
|---------------|----------------------------|------------|------------------|-------------------------------|---------------------------------|
| MLE           | $-\ell(\theta)$            | $O(n^{-1})$| None             | Any parametric model          | Poor at small $n$, overfits     |
| Ridge ($L_2$) | $-\ell(\theta)+\lambda\|\theta\|^2$ | $\lambda$-dependent| $\lambda$           | Linear, small-moderate $n$     | Needs tuning, unreliable in nonlinear settings |
| ICE           | $-\ell^*(\theta)$          | $O(n^{-3/2})$| None             | Any twice-differentiable model | Data-driven, no tuning required, robust at small-moderate $n$ |

MLE is optimistically biased, particularly in highly-parameterized or small-sample regimes. Ridge regularization requires hyper-parameter selection, commonly via cross-validation, and is less robust for non-Gaussian or highly nonlinear models. ICE directly removes $O(n^{-1})$ generalization bias, achieving $O(n^{-3/2})$ bias without requiring model-specific hyper-parameters, and derives its correction entirely from the data.

## 7. Practical Deployment and Usage Guidelines

ICE is suitable for scenarios where the prevention of overfitting is paramount and hyper-parameter free operation is desired:

- Particularly advantageous for small to moderate sample sizes ($n < 10^4$) and models with many parameters ($d > 50$).
- In distributed frameworks (e.g., Spark ML, via `setUseICE(true)`), ICE is drop-in and adds one additional per-iteration pass to accumulate diagonal Hessians.
- In custom implementations, a single backward pass should accumulate both gradient and diagonal Hessian per parameter, and apply the ICE correction within the loss function.
- For extremely small $n$, diagonal approximation may be restrictive; block-diagonal or low-rank alternatives can be explored but with increased computational cost.
- ICE integrates less naturally with explicit regularization such as dropout or $L_1/L_2$ penalties due to potential double penalization effects.

ICE is statistically robust in real-world distributed neural architectures, requires minimal code changes, incurs negligible memory overhead, and is validated as a practical alternative to unregularized likelihood optimization wherever model selection or out-of-sample accuracy is critical [1803.04947][2007.06157].

Source: https://www.emergentmind.com/topics/information-corrected-estimator-ice