---
title: Mallows Model Averaging
url: https://www.emergentmind.com/topics/mallows-model-averaging
type: topic
---

# Mallows Model Averaging

Searching arXiv for recent and foundational papers on Mallows model averaging and closely related extensions.
arXiv search query: all:"Mallows model averaging" OR ti:"Mallows model averaging"
Mallows model averaging denotes a class of frequentist model averaging procedures in which candidate estimators are combined by weights chosen through minimization of a Mallows \(C_p\)-type criterion. In its classical regression form, the method replaces single-model selection by a convex combination of candidate least-squares fits, with the central theoretical objective that the data-driven aggregate perform asymptotically as well as the best but infeasible averaged estimator under squared-error risk [2309.13239]. Subsequent work extends the same principle to generalized least squares with time-series dependence, misspecified models with location bias, robust convex losses, partially linear functional score models, and spatially varying coefficient models [1503.06401], [1912.01194], [1910.12210], [2105.00953], [2603.13930]. The collected literature also contains a distinct Mallows tradition for ranking data, where “Mallows” refers not to Mallows \(C_p\) but to a probability model on permutations; that usage is conceptually separate from regression model averaging [1902.08432].

## 1. Classical regression formulation

In the standard linear-regression setting, Mallows model averaging starts from a family of candidate models \(\mathcal{M}=\{\mathcal{I}_1,\dots,\mathcal{I}_{M_n}\}\), each producing a least-squares fitted value
\[
\hat f_{m|\mathcal M}=P_{m|\mathcal M}y, \qquad P_{m|\mathcal M}=X_{\mathcal I_m}(X_{\mathcal I_m}^\top X_{\mathcal I_m})^{-1}X_{\mathcal I_m}^\top.
\]
For a weight vector \(w=(w_1,\dots,w_{M_n})^\top\) in the simplex
\[
\mathcal W_{M_n}=\Bigl\{w\in[0,1]^{M_n}:\sum_{m=1}^{M_n}w_m=1\Bigr\},
\]
the averaged estimator is
\[
\hat f_{w|\mathcal M}=\sum_{m=1}^{M_n} w_m \hat f_{m|\mathcal M} = P_{w|\mathcal M}y.
\]
The Mallows criterion used to choose weights is
\[
C_n(w|\mathcal M,y) = \frac{1}{n}\|y-\hat f_{w|\mathcal M}\|^2+\frac{2\sigma^2}{n}k^\top w, \qquad k_m=|\mathcal I_m|,
\]
and the associated risk is
\[
R_n(w|\mathcal M,f) = \frac{1}{n}\|f-P_{w|\mathcal M}f\|^2 +\frac{\sigma^2}{n}\operatorname{tr}\!\left(P_{w|\mathcal M}^\top P_{w|\mathcal M}\right).
\]
A central property is that, if \(\sigma^2\) were known, \(C_n(w|\mathcal M,y)\) is an unbiased estimate of the model-averaging risk up to an additive constant:
\[
\mathbb E\, C_n(w|\mathcal M,y) = R_n(w|\mathcal M,f)+\sigma^2.
\]
The resulting Mallows model averaging estimator is obtained from
\[
\hat w|\mathcal M=\arg\min_{w\in\mathcal W_{M_n}}C_n(w|\mathcal M,y),
\]
which formalizes the idea that weight estimation is driven by an empirical proxy for prediction risk rather than by model identification [2309.13239].

This formulation is the direct descendant of Hansen’s 2007 MMA framework, which is repeatedly used in the later literature as the baseline reference point. In that tradition, MMA is a procedure for averaging estimators from multiple regression models, not a method for estimating a single “best” specification. A recurrent theme is that the candidate set may be misspecified, and that the true model need not belong to the candidate family.

## 2. Oracle optimality and the geometry of the weight space

The main theoretical justification for MMA is asymptotic optimality. In the formulation used in later theory, MMA is asymptotically optimal if
\[
Q_n(\hat w|\mathcal M,f) \le [1+o(1)]\min_{w\in\mathcal W}R_n(w|\mathcal M,f),
\]
so that the feasible data-driven aggregate asymptotically matches the best infeasible convex combination [2309.13239]. Earlier proofs often relied either on a discrete weight set
\[
\mathcal W_{|\mathcal M|}(N)
=
\left\{w:\sum_m w_m=1,\; w_m\in\{0,1/N,\dots,1\}\right\},
\]
or on strong restrictions on the candidate model collection. Later work shows that these restrictions can be harmful: in the nested setting, the optimal risk of model averaging may become an unreachable target under coarse discrete weighting, and in the non-nested setting a restricted candidate set can exclude the true optimal model size [2309.13239].

This issue appears particularly clearly in the time-series extension. For regression with infinitely many parameters and serially correlated errors, the feasible autocovariance-corrected Mallows procedure minimizes
\[
\hat C_n^*(w) = \bigl(Y_n-\hat\mu_n^{F}(w)\bigr)'\hat\Sigma_n^{-1}\bigl(Y_n-\hat\mu_n^{F}(w)\bigr) + 2\sum_{m=1}^M w_m k_m,
\]
over a sparse continuous weight class
\[
H_N=\bigcup_{\ell=1}^N H^{(\ell)},
\]
where each \(H^{(\ell)}\) contains weight vectors with exactly \(\ell\) nonzero components, each nonzero weight bounded below by a small \(\delta>0\), and satisfying
\[
w_m\in[0,1],\qquad \sum_{m=1}^M w_m=1.
\]
That continuous set includes Hansen’s discrete set as a proper subset,
\[
H_n(N)\subsetneq H_N,
\]
and Theorem 2 establishes
\[
\frac{L_n^F(\hat w_n)}{\inf_{w\in H_N}L_n^*(w)}\xrightarrow{p}1.
\]
The result is an explicit demonstration that continuous weighting can recover the oracle generalized squared-error performance while allowing more flexible approximation to the best averaging rule [1503.06401].

The broader theoretical literature sharpens the same point. In nested models, continuous-weight MMA satisfies asymptotic optimality under mild conditions, and the same line of work gives sufficient conditions for “full AOP” relative to the full nested model set \(\mathcal M_A\). In non-nested settings, a sufficient condition and a negative result are both available: with adequate ordering structure the oracle convex-combination risk is achievable, whereas without order information the ideal all-subset target is unattainable by any method on the stated classes [2309.13239].

## 3. Major extensions of the Mallows criterion

The classical least-squares criterion has been adapted to a wide range of settings in which the original MMA formulation is inadequate. The time-series case is foundational. When errors are serially dependent, efficient estimation within each candidate model is based on GLS rather than OLS, and the loss becomes the generalized squared error
\[
L_n^*(w)=\bigl(\hat\mu_n(w)-U_n\bigr)'\Sigma_n^{-1}\bigl(\hat\mu_n(w)-U_n\bigr).
\]
Because \(\Sigma_n^{-1}\) is unknown, a feasible procedure uses FGLS estimators built from a consistent inverse-covariance estimator obtained by banded Cholesky decomposition, and the resulting feasible autocovariance-corrected Mallows criterion is asymptotically efficient relative to the best continuous-weight GLS average [1503.06401].

In misspecified least-squares regression, mean-shift least squares model averaging enlarges the classical simplex-based averaging rule by adding a common intercept:
\[
\hat\mu(W,\alpha)=\alpha+\sum_{m=1}^M w_m P_mY.
\]
Its criterion,
\[
C_n(W,\alpha) = \bigl(y-\hat\mu(W)-\alpha\mathbf 1\bigr)^\top \bigl(y-\hat\mu(W)-\alpha\mathbf 1\bigr) +2\sigma^2 K^\top W,
\]
is designed to control both location bias and regression error through a common constant. The corresponding estimator is asymptotically optimal over the enlarged parameter space \((W,\alpha)\), and the construction weakly dominates the original no-shift MMA objective because \(\alpha=0\) is admissible [1912.01194].

Robust Mallows-type averaging replaces squared loss by a general convex loss \(\rho\), with weight choice based on an approximate expected prediction error rather than the expected squared error. For fixed design, the criterion is
\[
C_n(\boldsymbol{w}) = \sum_{i=1}^n \rho\!\left(\varepsilon_i(\boldsymbol{w})\right) + C_\rho \sum_{m=1}^M w_m k_m,
\]
and for random design,
\[
\widetilde{C}_n(\boldsymbol{w}) = \sum_{i=1}^n \rho\!\left(\varepsilon_i(\boldsymbol{w})\right) + \sum_{m=1}^M w_m k_m C_{\rho(m)}.
\]
The paper explicitly studies least absolute deviation and Huber loss, emphasizing that large residuals are downweighted relative to least squares and that classical MMA can deteriorate badly in contaminated samples [1910.12210].

The functional-data extension addresses partially linear functional score models, where the response depends on scalar covariates and on a nonparametric effect of estimated transformed FPC scores. There the feasible Mallows-type criterion is
\[
\hat C_n(\omega)=\|Y-\hat\mu(\omega)\|^2+2\,\operatorname{tr}\!\big(\hat\Omega\,H(\omega)\big),
\]
with weights chosen over the simplex
\[
\mathcal H_n=\left\{\omega=(\omega_1,\ldots,\omega_M)^\top:\ \omega_m\ge 0,\ \sum_{m=1}^M\omega_m=1\right\}.
\]
The resulting estimator is proved asymptotically optimal even though the nonparametric covariates are latent FPC scores that must themselves be estimated from noisy functional observations [2105.00953].

The spatial extension develops the spatially varying coefficient Mallows model averaging estimator
\[
\widehat{\bm{\mu}(\mathbf{w})} = \sum_{m=1}^{M} w_m \mathbf{P}_{(m)}\mathbf{Y} = \mathbf{P}(\mathbf{w})\mathbf{Y},
\]
with feasible criterion
\[
\widehat{C}_n(\mathbf{w}) = \|\mathbf{Y}-\widehat{\bm{\mu}(\mathbf{w})}\|^2 + 2\operatorname{tr}(\mathbf{P}(\mathbf{w})\widehat{\bm{\Omega}}).
\]
In the all-misspecified regime, the method achieves
\[
\frac{L_n(\widetilde{\mathbf{w}})}{\inf_{\mathbf{w}\in \mathcal{H}_n} L_n(\mathbf{w})} \overset{p}{\rightarrow} 1,
\]
while in the quasi-correct regime the weights asymptotically concentrate on quasi-correct models and the estimator is consistent for the true conditional mean [2603.13930].

A separate, closely related line arises for MACML-estimated multinomial probit models. That work does not derive the classical Mallows \(C_p\) formula directly, but it develops an asymptotically MSE-optimal averaging scheme of Mallows-like spirit, with weights solving
\[
\hat{w}_{mse} = \arg \min_{w \in \mathbb{R}^{M},\ \sum_{m=1}^M w_m = 1} w'Fw.
\]
It therefore belongs to the broader frequentist model-averaging family even though the paper explicitly distinguishes it from textbook MMA [1704.00183].

## 4. Finite-sample theory and the all-subset problem

A major recent development is the shift from purely asymptotic oracle statements to non-asymptotic analysis. For least-squares model averaging over an arbitrary candidate set \(\mathcal M\), the Mallows-type criterion
\[
C_{n}(\bw|\mathcal{M},\lambda)\triangleq n^{-1}\left\|\by-\bP(\bw)\by\right\|^{2}+2\lambda^2{\sigma}^{2}\tr\bP(\bw)
\]
admits several oracle inequalities under only finite fourth moments of the errors [2505.02637]. The classical MMA choice corresponds to
\[
\lambda_1=\sqrt{1/n},
\]
with
\[
{\bw}_1\triangleq\arg\min_{\bw \in \mathcal{W}} C_{n}(\bw|\mathcal{M}, \lambda_1).
\]
The sharp oracle inequality and the non-exact oracle inequality given there imply asymptotic optimality under milder conditions than earlier loss-based results and yield faster excess-risk control in low-to-moderate candidate-set regimes [2505.02637].

The same paper addresses the all-subset combination problem, which asks how closely one can approach the risk of the ideal model average over all subsets of regressors. In the orthogonal-basis formulation,
\[
\mathcal{M}_{AS}\triangleq \{ \mathcal{I} : \mathcal{I} \subseteq \{1,\ldots,p\} \},
\]
the optimal all-subset model-averaging risk is
\[
R_n\!\left({\bmu}_{\bw^*|\mathcal{M}_{AS}}, \bmu\right) = \sum_{j=1}^{p}\frac{\theta_j^2 \sigma^2/n}{\theta_j^2+ \sigma^2/n}.
\]
Yet there is a fundamental limit to how well any estimator can match that target uniformly. With Gaussian noise and parameter spaces containing the “hardest cube,” the minimax risk ratio is greater than \(2\) when \(p\) is fixed and large enough, and at least \([1+o(1)]\,2\log p\) when \(p\to\infty\) [2505.02637].

To attain that lower-bound rate, the dimension-adaptive \(C_p\) procedure uses only the univariate candidate models
\[
\mathcal{M}_U \triangleq \{\{1\},\ldots,\{p\}\},
\]
with tuning parameter
\[
\lambda_2 \triangleq \sqrt{\frac{2\log p}{n}},
\]
and solves
\[
{\bw}_{2} \triangleq \argmin_{\bw \in \mathcal{H}} \left\{ n^{-1}\Bigl\| \by - \sum_{j=1}^{p}w_j {\bmu}_{j} \Bigr\|^2 + 2\lambda^2_{2} \sigma^2 \bw^{\top}\boldsymbol{1} \right\}.
\]
The optimizer is explicit:
\[
{w}_{2j}=\left(1-\frac{\lambda_2^2\sigma^2}{\tilde{\theta}_j^2}\right)_+.
\]
This estimator attains the minimax lower-bound rate relative to the optimal all-subset MA risk and links Mallows-type averaging directly to nonnegative-garrote-type shrinkage [2505.02637].

These results refine the earlier achievability theory. They suggest that the statistical difficulty of model averaging is not confined to weight estimation; candidate-set design is itself a primary theoretical object. A plausible implication is that the effectiveness of MMA depends jointly on the criterion and on how the model family encodes ordering, sparsity, or block structure.

## 5. Shrinkage, Stein rules, and the relation to model selection

A second major reinterpretation views Mallows model averaging as a shrinkage estimator. In nested linear regression, define the orthogonal increments
\[
D_{m\mid\mathcal M}=P_{k_m}-P_{k_{m-1}},\qquad m=1,\ldots,M_n,
\]
with \(P_{k_0}=0\), and corresponding block components
\[
y_{m\mid\mathcal M}=D_{m\mid\mathcal M}y=\mu_{m\mid\mathcal M}+\varepsilon_{m\mid\mathcal M}.
\]
Then the model-averaged estimator can be written as
\[
\hat\mu_{w\mid\mathcal M}=\sum_{m=1}^{M_n}\gamma_m y_{m\mid\mathcal M},
\]
where \(\gamma_m=\sum_{j=m}^{M_n}w_j\). The simplex constraint on \(w\) is equivalent to monotonicity of the cumulative shrinkage weights,
\[
1=\gamma_1\ge \gamma_2\ge\cdots\ge\gamma_{M_n}\ge 0,
\]
so the optimal MA estimator is the best linear estimator with monotonically non-increasing weights in the Gaussian sequence model [2309.14596].

Under a relaxed weight set, minimizing the same Mallows-type criterion yields the explicit positive-part Stein rule
\[
\gamma_{\text{\scriptsize STE},m} =\left(1-\frac{\sigma^2_{m\mid\mathcal M}}{\|y_{m\mid\mathcal M}\|^2}\right)_+,
\]
and therefore
\[
\hat\mu_{w_{\text{\tiny STE}\mid\mathcal M}} =\sum_{m=1}^{M_n}\left(1-\frac{\sigma^2_{m\mid\mathcal M}}{\|y_{m\mid\mathcal M}\|^2}\right)_+y_{m\mid\mathcal M}.
\]
The same work develops a penalized blockwise Stein estimator
\[
\gamma_m=\left(1-\frac{\sigma^2_{m\mid\mathcal M}(1+\varphi_m)}{\|y_{m\mid\mathcal M}\|^2}\right)_+,
\]
which is asymptotically optimal over a broad parameter space when the variance is known [2309.14596].

This shrinkage perspective clarifies why model averaging and model selection are closely related but not identical. The all-subset theory shows that soft-thresholding and hard-thresholding can achieve the same minimax \(2\log p\) rate relative to the optimal all-subset MA risk in the orthogonal setting, revealing what the paper calls the implicit ensembling effects of several model-selection procedures [2505.02637]. At the same time, the shrinkage analysis shows that the simplex constraint can reduce model averaging’s potential when blockwise signal-to-noise ratios are not monotonically ordered, so that the relaxed Stein-type solution may improve on classical MMA [2309.14596].

This line of work has also altered the interpretation of asymptotic optimality. Rather than viewing MMA only as a convex-combination device, recent theory treats it as a structured shrinkage rule whose admissible geometry is encoded by the candidate model sequence.

## 6. Mallows \(C_p\) averaging and Mallows rank models

The literature surveyed here suggests that the phrase “Mallows model averaging” is terminologically overloaded. In regression and econometrics, “Mallows” refers to Mallows \(C_p\) and to weight selection by risk-unbiased or asymptotically risk-unbiased criteria. In ranking and preference learning, “Mallows” refers instead to a distance-based probability model on permutations.

The Bayesian Mallows ranking framework has likelihood
\[
P(\mathbf r \mid \alpha,\boldsymbol\rho) = \frac{1}{Z_n(\alpha)} \exp\!\left[-\frac{\alpha}{n}\, d(\mathbf r,\boldsymbol\rho)\right] \,1_{\mathcal P_n}(\mathbf r),
\]
with consensus ranking \(\boldsymbol\rho\), concentration parameter \(\alpha\), and right-invariant distance \(d\). BayesMallows provides MCMC-based posterior inference, finite mixtures, support for partial rankings and pairwise comparisons, and posterior summaries such as MAP and CP consensus rankings [1902.08432]. Mixed Membership Mallows Models generalize mixture-of-Mallows by allowing each user to express a probabilistic combination of shared latent Mallows components, learned through a topic-model analogy for pairwise comparisons [1504.00757]. Selective Mallows generalizes the classical ranking model to incomplete rankings on arbitrary subsets and establishes asymptotically tight sample-complexity bounds for recovering the latent full ranking or the top-\(k\) items [2011.00810]. Partition-Mallows aggregation combines a partition into relevant and background entities with a Mallows model on the relevant subset, thereby jointly estimating relevance, internal ordering, and ranker reliability [2104.07261]. Pseudo-Mallows replaces expensive MCMC posterior sampling by a sequentially constrained variational approximation to the Bayesian Mallows posterior on permutations [2205.13911].

These ranking models are aggregation procedures in the sense that they produce a consensus ranking or average over posterior ranking uncertainty, and some papers informally describe that process as “averaging” preferences or latent components. However, they are not Mallows \(C_p\)-based model averaging procedures. The common word “Mallows” designates different mathematical objects: a prediction-risk criterion in regression on the one hand, and a distribution on rankings on the other. A plausible implication is that the intended meaning of “Mallows model averaging” must always be inferred from the surrounding domain—regression aggregation, preference aggregation, or permutation modeling—rather than from the phrase alone.

Source: https://www.emergentmind.com/topics/mallows-model-averaging