Mallows Model Averaging

• Mallows model averaging is a frequentist procedure that combines candidate estimators using convex weights determined by minimizing a Mallows Cₚ-type criterion.
• It achieves asymptotic oracle optimality and extends to settings such as time-series, misspecified models, robust losses, and spatial analysis.
• Recent advances integrate shrinkage interpretations, finite-sample guarantees, and generalized loss functions to enhance practical regression performance.

Searching arXiv for recent and foundational papers on Mallows model averaging and closely related extensions. arXiv search query: all:"Mallows model averaging" OR ti:"Mallows model averaging" Mallows model averaging denotes a class of frequentist model averaging procedures in which candidate estimators are combined by weights chosen through minimization of a Mallows $C_p$-type criterion. In its classical regression form, the method replaces single-model selection by a convex combination of candidate least-squares fits, with the central theoretical objective that the data-driven aggregate perform asymptotically as well as the best but infeasible averaged estimator under squared-error risk (Peng et al., 2023). Subsequent work extends the same principle to generalized least squares with time-series dependence, misspecified models with location bias, robust convex losses, partially linear functional score models, and spatially varying coefficient models (Cheng et al., 2015, McAlinn et al., 2019, Wang et al., 2019, Liu et al., 2021, Yong et al., 14 Mar 2026). The collected literature also contains a distinct Mallows tradition for ranking data, where “Mallows” refers not to Mallows $C_p$ but to a probability model on permutations; that usage is conceptually separate from regression model averaging (Sørensen et al., 2019).

1. Classical regression formulation

In the standard linear-regression setting, Mallows model averaging starts from a family of candidate models $$\mathcal{M}=\{\mathcal{I}_1,\dots,\mathcal{I}_{M_n}\}$$, each producing a least-squares fitted value

$$\hat f_{m|\mathcal M}=P_{m|\mathcal M}y, \qquad P_{m|\mathcal M}=X_{\mathcal I_m}(X_{\mathcal I_m}^\top X_{\mathcal I_m})^{-1}X_{\mathcal I_m}^\top.$$

For a weight vector $w=(w_1,\dots,w_{M_n})^\top$ in the simplex

$$\mathcal W_{M_n}=\Bigl\{w\in[0,1]^{M_n}:\sum_{m=1}^{M_n}w_m=1\Bigr\},$$

the averaged estimator is

$$\hat f_{w|\mathcal M}=\sum_{m=1}^{M_n} w_m \hat f_{m|\mathcal M} = P_{w|\mathcal M}y.$$

The Mallows criterion used to choose weights is

$$C_n(w|\mathcal M,y) = \frac{1}{n}\|y-\hat f_{w|\mathcal M}\|^2+\frac{2\sigma^2}{n}k^\top w, \qquad k_m=|\mathcal I_m|,$$

and the associated risk is

$$R_n(w|\mathcal M,f) = \frac{1}{n}\|f-P_{w|\mathcal M}f\|^2 +\frac{\sigma^2}{n}\operatorname{tr}\!\left(P_{w|\mathcal M}^\top P_{w|\mathcal M}\right).$$

A central property is that, if $\sigma^2$ were known, $C_p$0 is an unbiased estimate of the model-averaging risk up to an additive constant: $C_p$1 The resulting Mallows model averaging estimator is obtained from

$C_p$2

which formalizes the idea that weight estimation is driven by an empirical proxy for prediction risk rather than by model identification (Peng et al., 2023).

This formulation is the direct descendant of Hansen’s 2007 MMA framework, which is repeatedly used in the later literature as the baseline reference point. In that tradition, MMA is a procedure for averaging estimators from multiple regression models, not a method for estimating a single “best” specification. A recurrent theme is that the candidate set may be misspecified, and that the true model need not belong to the candidate family.

2. Oracle optimality and the geometry of the weight space

The main theoretical justification for MMA is asymptotic optimality. In the formulation used in later theory, MMA is asymptotically optimal if

$C_p$3

so that the feasible data-driven aggregate asymptotically matches the best infeasible convex combination (Peng et al., 2023). Earlier proofs often relied either on a discrete weight set

$C_p$4

or on strong restrictions on the candidate model collection. Later work shows that these restrictions can be harmful: in the nested setting, the optimal risk of model averaging may become an unreachable target under coarse discrete weighting, and in the non-nested setting a restricted candidate set can exclude the true optimal model size (Peng et al., 2023).

This issue appears particularly clearly in the time-series extension. For regression with infinitely many parameters and serially correlated errors, the feasible autocovariance-corrected Mallows procedure minimizes

$C_p$5

over a sparse continuous weight class

$C_p$6

where each $C_p$7 contains weight vectors with exactly $C_p$8 nonzero components, each nonzero weight bounded below by a small $C_p$9, and satisfying

$$\mathcal{M}=\{\mathcal{I}_1,\dots,\mathcal{I}_{M_n}\}$$0

That continuous set includes Hansen’s discrete set as a proper subset,

$$\mathcal{M}=\{\mathcal{I}_1,\dots,\mathcal{I}_{M_n}\}$$1

and Theorem 2 establishes

$$\mathcal{M}=\{\mathcal{I}_1,\dots,\mathcal{I}_{M_n}\}$$2

The result is an explicit demonstration that continuous weighting can recover the oracle generalized squared-error performance while allowing more flexible approximation to the best averaging rule (Cheng et al., 2015).

The broader theoretical literature sharpens the same point. In nested models, continuous-weight MMA satisfies asymptotic optimality under mild conditions, and the same line of work gives sufficient conditions for “full AOP” relative to the full nested model set $$\mathcal{M}=\{\mathcal{I}_1,\dots,\mathcal{I}_{M_n}\}$$3. In non-nested settings, a sufficient condition and a negative result are both available: with adequate ordering structure the oracle convex-combination risk is achievable, whereas without order information the ideal all-subset target is unattainable by any method on the stated classes (Peng et al., 2023).

3. Major extensions of the Mallows criterion

The classical least-squares criterion has been adapted to a wide range of settings in which the original MMA formulation is inadequate. The time-series case is foundational. When errors are serially dependent, efficient estimation within each candidate model is based on GLS rather than OLS, and the loss becomes the generalized squared error

$$\mathcal{M}=\{\mathcal{I}_1,\dots,\mathcal{I}_{M_n}\}$$4

Because $$\mathcal{M}=\{\mathcal{I}_1,\dots,\mathcal{I}_{M_n}\}$$5 is unknown, a feasible procedure uses FGLS estimators built from a consistent inverse-covariance estimator obtained by banded Cholesky decomposition, and the resulting feasible autocovariance-corrected Mallows criterion is asymptotically efficient relative to the best continuous-weight GLS average (Cheng et al., 2015).

In misspecified least-squares regression, mean-shift least squares model averaging enlarges the classical simplex-based averaging rule by adding a common intercept: $$\mathcal{M}=\{\mathcal{I}_1,\dots,\mathcal{I}_{M_n}\}$$6 Its criterion,

$$\mathcal{M}=\{\mathcal{I}_1,\dots,\mathcal{I}_{M_n}\}$$7

is designed to control both location bias and regression error through a common constant. The corresponding estimator is asymptotically optimal over the enlarged parameter space $$\mathcal{M}=\{\mathcal{I}_1,\dots,\mathcal{I}_{M_n}\}$$8, and the construction weakly dominates the original no-shift MMA objective because $$\mathcal{M}=\{\mathcal{I}_1,\dots,\mathcal{I}_{M_n}\}$$9 is admissible (McAlinn et al., 2019).

Robust Mallows-type averaging replaces squared loss by a general convex loss $$\hat f_{m|\mathcal M}=P_{m|\mathcal M}y, \qquad P_{m|\mathcal M}=X_{\mathcal I_m}(X_{\mathcal I_m}^\top X_{\mathcal I_m})^{-1}X_{\mathcal I_m}^\top.$$0, with weight choice based on an approximate expected prediction error rather than the expected squared error. For fixed design, the criterion is

$$\hat f_{m|\mathcal M}=P_{m|\mathcal M}y, \qquad P_{m|\mathcal M}=X_{\mathcal I_m}(X_{\mathcal I_m}^\top X_{\mathcal I_m})^{-1}X_{\mathcal I_m}^\top.$$1

and for random design,

$$\hat f_{m|\mathcal M}=P_{m|\mathcal M}y, \qquad P_{m|\mathcal M}=X_{\mathcal I_m}(X_{\mathcal I_m}^\top X_{\mathcal I_m})^{-1}X_{\mathcal I_m}^\top.$$2

The paper explicitly studies least absolute deviation and Huber loss, emphasizing that large residuals are downweighted relative to least squares and that classical MMA can deteriorate badly in contaminated samples (Wang et al., 2019).

The functional-data extension addresses partially linear functional score models, where the response depends on scalar covariates and on a nonparametric effect of estimated transformed FPC scores. There the feasible Mallows-type criterion is

$$\hat f_{m|\mathcal M}=P_{m|\mathcal M}y, \qquad P_{m|\mathcal M}=X_{\mathcal I_m}(X_{\mathcal I_m}^\top X_{\mathcal I_m})^{-1}X_{\mathcal I_m}^\top.$$3

with weights chosen over the simplex

$$\hat f_{m|\mathcal M}=P_{m|\mathcal M}y, \qquad P_{m|\mathcal M}=X_{\mathcal I_m}(X_{\mathcal I_m}^\top X_{\mathcal I_m})^{-1}X_{\mathcal I_m}^\top.$$4

The resulting estimator is proved asymptotically optimal even though the nonparametric covariates are latent FPC scores that must themselves be estimated from noisy functional observations (Liu et al., 2021).

The spatial extension develops the spatially varying coefficient Mallows model averaging estimator

$$\hat f_{m|\mathcal M}=P_{m|\mathcal M}y, \qquad P_{m|\mathcal M}=X_{\mathcal I_m}(X_{\mathcal I_m}^\top X_{\mathcal I_m})^{-1}X_{\mathcal I_m}^\top.$$5

with feasible criterion

$$\hat f_{m|\mathcal M}=P_{m|\mathcal M}y, \qquad P_{m|\mathcal M}=X_{\mathcal I_m}(X_{\mathcal I_m}^\top X_{\mathcal I_m})^{-1}X_{\mathcal I_m}^\top.$$6

In the all-misspecified regime, the method achieves

$$\hat f_{m|\mathcal M}=P_{m|\mathcal M}y, \qquad P_{m|\mathcal M}=X_{\mathcal I_m}(X_{\mathcal I_m}^\top X_{\mathcal I_m})^{-1}X_{\mathcal I_m}^\top.$$7

while in the quasi-correct regime the weights asymptotically concentrate on quasi-correct models and the estimator is consistent for the true conditional mean (Yong et al., 14 Mar 2026).

A separate, closely related line arises for MACML-estimated multinomial probit models. That work does not derive the classical Mallows $$\hat f_{m|\mathcal M}=P_{m|\mathcal M}y, \qquad P_{m|\mathcal M}=X_{\mathcal I_m}(X_{\mathcal I_m}^\top X_{\mathcal I_m})^{-1}X_{\mathcal I_m}^\top.$$8 formula directly, but it develops an asymptotically MSE-optimal averaging scheme of Mallows-like spirit, with weights solving

$$\hat f_{m|\mathcal M}=P_{m|\mathcal M}y, \qquad P_{m|\mathcal M}=X_{\mathcal I_m}(X_{\mathcal I_m}^\top X_{\mathcal I_m})^{-1}X_{\mathcal I_m}^\top.$$9

It therefore belongs to the broader frequentist model-averaging family even though the paper explicitly distinguishes it from textbook MMA (Batram et al., 2017).

4. Finite-sample theory and the all-subset problem

A major recent development is the shift from purely asymptotic oracle statements to non-asymptotic analysis. For least-squares model averaging over an arbitrary candidate set $w=(w_1,\dots,w_{M_n})^\top$0, the Mallows-type criterion

$w=(w_1,\dots,w_{M_n})^\top$1

admits several oracle inequalities under only finite fourth moments of the errors (Peng, 5 May 2025). The classical MMA choice corresponds to

$w=(w_1,\dots,w_{M_n})^\top$2

with

$w=(w_1,\dots,w_{M_n})^\top$3

The sharp oracle inequality and the non-exact oracle inequality given there imply asymptotic optimality under milder conditions than earlier loss-based results and yield faster excess-risk control in low-to-moderate candidate-set regimes (Peng, 5 May 2025).

The same paper addresses the all-subset combination problem, which asks how closely one can approach the risk of the ideal model average over all subsets of regressors. In the orthogonal-basis formulation,

$w=(w_1,\dots,w_{M_n})^\top$4

the optimal all-subset model-averaging risk is

$w=(w_1,\dots,w_{M_n})^\top$5

Yet there is a fundamental limit to how well any estimator can match that target uniformly. With Gaussian noise and parameter spaces containing the “hardest cube,” the minimax risk ratio is greater than $w=(w_1,\dots,w_{M_n})^\top$6 when $w=(w_1,\dots,w_{M_n})^\top$7 is fixed and large enough, and at least $w=(w_1,\dots,w_{M_n})^\top$8 when $w=(w_1,\dots,w_{M_n})^\top$9 (Peng, 5 May 2025).

To attain that lower-bound rate, the dimension-adaptive $$\mathcal W_{M_n}=\Bigl\{w\in[0,1]^{M_n}:\sum_{m=1}^{M_n}w_m=1\Bigr\},$$0 procedure uses only the univariate candidate models

$$\mathcal W_{M_n}=\Bigl\{w\in[0,1]^{M_n}:\sum_{m=1}^{M_n}w_m=1\Bigr\},$$1

with tuning parameter

$$\mathcal W_{M_n}=\Bigl\{w\in[0,1]^{M_n}:\sum_{m=1}^{M_n}w_m=1\Bigr\},$$2

and solves

$$\mathcal W_{M_n}=\Bigl\{w\in[0,1]^{M_n}:\sum_{m=1}^{M_n}w_m=1\Bigr\},$$3

The optimizer is explicit: $$\mathcal W_{M_n}=\Bigl\{w\in[0,1]^{M_n}:\sum_{m=1}^{M_n}w_m=1\Bigr\},$$4 This estimator attains the minimax lower-bound rate relative to the optimal all-subset MA risk and links Mallows-type averaging directly to nonnegative-garrote-type shrinkage (Peng, 5 May 2025).

These results refine the earlier achievability theory. They suggest that the statistical difficulty of model averaging is not confined to weight estimation; candidate-set design is itself a primary theoretical object. A plausible implication is that the effectiveness of MMA depends jointly on the criterion and on how the model family encodes ordering, sparsity, or block structure.

5. Shrinkage, Stein rules, and the relation to model selection

A second major reinterpretation views Mallows model averaging as a shrinkage estimator. In nested linear regression, define the orthogonal increments

$$\mathcal W_{M_n}=\Bigl\{w\in[0,1]^{M_n}:\sum_{m=1}^{M_n}w_m=1\Bigr\},$$5

with $$\mathcal W_{M_n}=\Bigl\{w\in[0,1]^{M_n}:\sum_{m=1}^{M_n}w_m=1\Bigr\},$$6, and corresponding block components

$$\mathcal W_{M_n}=\Bigl\{w\in[0,1]^{M_n}:\sum_{m=1}^{M_n}w_m=1\Bigr\},$$7

Then the model-averaged estimator can be written as

$$\mathcal W_{M_n}=\Bigl\{w\in[0,1]^{M_n}:\sum_{m=1}^{M_n}w_m=1\Bigr\},$$8

where $$\mathcal W_{M_n}=\Bigl\{w\in[0,1]^{M_n}:\sum_{m=1}^{M_n}w_m=1\Bigr\},$$9. The simplex constraint on $$\hat f_{w|\mathcal M}=\sum_{m=1}^{M_n} w_m \hat f_{m|\mathcal M} = P_{w|\mathcal M}y.$$0 is equivalent to monotonicity of the cumulative shrinkage weights,

$$\hat f_{w|\mathcal M}=\sum_{m=1}^{M_n} w_m \hat f_{m|\mathcal M} = P_{w|\mathcal M}y.$$1

so the optimal MA estimator is the best linear estimator with monotonically non-increasing weights in the Gaussian sequence model (Peng, 2023).

Under a relaxed weight set, minimizing the same Mallows-type criterion yields the explicit positive-part Stein rule

$$\hat f_{w|\mathcal M}=\sum_{m=1}^{M_n} w_m \hat f_{m|\mathcal M} = P_{w|\mathcal M}y.$$2

and therefore

$$\hat f_{w|\mathcal M}=\sum_{m=1}^{M_n} w_m \hat f_{m|\mathcal M} = P_{w|\mathcal M}y.$$3

The same work develops a penalized blockwise Stein estimator

$$\hat f_{w|\mathcal M}=\sum_{m=1}^{M_n} w_m \hat f_{m|\mathcal M} = P_{w|\mathcal M}y.$$4

which is asymptotically optimal over a broad parameter space when the variance is known (Peng, 2023).

This shrinkage perspective clarifies why model averaging and model selection are closely related but not identical. The all-subset theory shows that soft-thresholding and hard-thresholding can achieve the same minimax $$\hat f_{w|\mathcal M}=\sum_{m=1}^{M_n} w_m \hat f_{m|\mathcal M} = P_{w|\mathcal M}y.$$5 rate relative to the optimal all-subset MA risk in the orthogonal setting, revealing what the paper calls the implicit ensembling effects of several model-selection procedures (Peng, 5 May 2025). At the same time, the shrinkage analysis shows that the simplex constraint can reduce model averaging’s potential when blockwise signal-to-noise ratios are not monotonically ordered, so that the relaxed Stein-type solution may improve on classical MMA (Peng, 2023).

This line of work has also altered the interpretation of asymptotic optimality. Rather than viewing MMA only as a convex-combination device, recent theory treats it as a structured shrinkage rule whose admissible geometry is encoded by the candidate model sequence.

6. Mallows $$\hat f_{w|\mathcal M}=\sum_{m=1}^{M_n} w_m \hat f_{m|\mathcal M} = P_{w|\mathcal M}y.$$6 averaging and Mallows rank models

The literature surveyed here suggests that the phrase “Mallows model averaging” is terminologically overloaded. In regression and econometrics, “Mallows” refers to Mallows $$\hat f_{w|\mathcal M}=\sum_{m=1}^{M_n} w_m \hat f_{m|\mathcal M} = P_{w|\mathcal M}y.$$7 and to weight selection by risk-unbiased or asymptotically risk-unbiased criteria. In ranking and preference learning, “Mallows” refers instead to a distance-based probability model on permutations.

The Bayesian Mallows ranking framework has likelihood

$$\hat f_{w|\mathcal M}=\sum_{m=1}^{M_n} w_m \hat f_{m|\mathcal M} = P_{w|\mathcal M}y.$$8

with consensus ranking $$\hat f_{w|\mathcal M}=\sum_{m=1}^{M_n} w_m \hat f_{m|\mathcal M} = P_{w|\mathcal M}y.$$9, concentration parameter $$C_n(w|\mathcal M,y) = \frac{1}{n}\|y-\hat f_{w|\mathcal M}\|^2+\frac{2\sigma^2}{n}k^\top w, \qquad k_m=|\mathcal I_m|,$$0, and right-invariant distance $$C_n(w|\mathcal M,y) = \frac{1}{n}\|y-\hat f_{w|\mathcal M}\|^2+\frac{2\sigma^2}{n}k^\top w, \qquad k_m=|\mathcal I_m|,$$1. BayesMallows provides MCMC-based posterior inference, finite mixtures, support for partial rankings and pairwise comparisons, and posterior summaries such as MAP and CP consensus rankings (Sørensen et al., 2019). Mixed Membership Mallows Models generalize mixture-of-Mallows by allowing each user to express a probabilistic combination of shared latent Mallows components, learned through a topic-model analogy for pairwise comparisons (Ding et al., 2015). Selective Mallows generalizes the classical ranking model to incomplete rankings on arbitrary subsets and establishes asymptotically tight sample-complexity bounds for recovering the latent full ranking or the top-$$C_n(w|\mathcal M,y) = \frac{1}{n}\|y-\hat f_{w|\mathcal M}\|^2+\frac{2\sigma^2}{n}k^\top w, \qquad k_m=|\mathcal I_m|,$$2 items (Fotakis et al., 2020). Partition-Mallows aggregation combines a partition into relevant and background entities with a Mallows model on the relevant subset, thereby jointly estimating relevance, internal ordering, and ranker reliability (Zhu et al., 2021). Pseudo-Mallows replaces expensive MCMC posterior sampling by a sequentially constrained variational approximation to the Bayesian Mallows posterior on permutations (Liu et al., 2022).

These ranking models are aggregation procedures in the sense that they produce a consensus ranking or average over posterior ranking uncertainty, and some papers informally describe that process as “averaging” preferences or latent components. However, they are not Mallows $$C_n(w|\mathcal M,y) = \frac{1}{n}\|y-\hat f_{w|\mathcal M}\|^2+\frac{2\sigma^2}{n}k^\top w, \qquad k_m=|\mathcal I_m|,$$3-based model averaging procedures. The common word “Mallows” designates different mathematical objects: a prediction-risk criterion in regression on the one hand, and a distribution on rankings on the other. A plausible implication is that the intended meaning of “Mallows model averaging” must always be inferred from the surrounding domain—regression aggregation, preference aggregation, or permutation modeling—rather than from the phrase alone.