Regression Mean (RegMean) Overview

• RegMean is defined as the conditional mean function or its approximation (e.g., a linear predictor) that underpins regression analysis.
• It extends to simultaneous mean–variance and robust regression techniques, providing consistent and efficient estimation even under model misspecification and heteroskedasticity.
• In neural networks, RegMean serves as a training-free, layerwise regression method for merging candidate models, enhancing computational efficiency and model explainability.

Regression Mean (RegMean) denotes a family of concepts centered on representing, approximating, or estimating a conditional mean. In its most standard statistical sense, the object is the regression mean function $m(x)=\mathbb{E}[Y\mid X=x]$, or a structured approximation such as a linear predictor $x'\beta$. In simultaneous mean–variance regression, RegMean is the location component $x'\beta^*$ chosen jointly with a scale function; in misspecification theory it is the conditional mean together with population functionals that summarize how changes in the distribution of covariates move $\mathbb{E}[Y]$; and in recent model-merging work, “RegMean” names a training-free layerwise linear-regression procedure for combining neural networks. This suggests that the term is context-dependent, but its common core is mean approximation under explicit structural constraints (Spady et al., 2018, Brannath et al., 2014, Nguyen et al., 5 Aug 2025).

1. Conditional-mean foundations

The most general object behind RegMean is the conditional mean function

$m(x)=\mathbb{E}[Y\mid X=x].$

In linear settings, this is approximated by

$m(x;\beta)=x'\beta,$

and the associated population target is often the least-squares projection

$$\beta=\arg\min_b \mathbb{E}\big[(Y-X^\top b)^2\big].$$

Under exact linearity, $x'\beta$ is the regression mean itself; under misspecification, it is the best linear approximation in an $L^2$ sense (Spady et al., 2018, Brannath et al., 2014).

A complementary interpretation replaces conditional-mean geometry by population perturbations of the covariate distribution. In that formulation, the “mean impact”

$$m_X(Y):=\sup_{\delta(X)\in L^2,\ \mathbb{E}[\delta(X)]=0,\ \mathbb{E}[\delta(X)^2]=1}\mathbb{E}[Y\delta(X)]$$

equals

$x'\beta$0

The same framework defines a linear mean impact

$x'\beta$1

which provides a conservative linear approximation to the nonlinear dependence of the regression mean on $x'\beta$2 (Brannath et al., 2014).

The same mean-centered viewpoint also appears in survival analysis, but with the target shifted from $x'\beta$3 to the mean residual life

$x'\beta$4

Here the relevant “regression mean” is time-indexed and conditional on survival up to $x'\beta$5, rather than being an ordinary one-shot conditional expectation (Poynor et al., 2014).

2. RegMean in simultaneous mean–variance regression

In simultaneous mean–variance regression (MVR), RegMean is the linear approximation

$x'\beta$6

obtained by minimizing a joint location–scale loss rather than ordinary quadratic loss. The model writes

$x'\beta$7

with $x'\beta$8 positive, strictly increasing, and convex. The population criterion is

$x'\beta$9

and $x'\beta^*$0 is the unique minimizer (Spady et al., 2018).

This construction changes the meaning of the regression mean. Under misspecification, $x'\beta^*$1 is not the unweighted best linear approximation to $x'\beta^*$2; it is the best weighted approximation once the scale approximation $x'\beta^*$3 is chosen jointly. The first-order condition

$x'\beta^*$4

makes the mean equation look like weighted least squares, but the weights $x'\beta^*$5 are not fixed exogenously. They arise endogenously from the companion scale condition

$x'\beta^*$6

so the location and variance approximations are determined simultaneously (Spady et al., 2018).

The paper emphasizes several consequences. The loss is globally convex and strictly convex over the parameter space, so the MVR solution exists and is unique even under general misspecification. If the mean model is correctly specified, then $x'\beta^*$7 regardless of scale misspecification, which gives a strong robustness property for RegMean. For the scale choices $x'\beta^*$8 and $x'\beta^*$9, the resulting location–scale approximation weakly dominates the OLS location model under a Kullback–Leibler measure of divergence, with strict improvement in the presence of heteroskedasticity. Estimation is consistent and asymptotically normal under misspecification, and the plug-in sandwich covariance $\mathbb{E}[Y]$0 is valid without additional heteroskedasticity corrections (Spady et al., 2018).

3. Misspecification, robustness, and heavy-tailed RegMean targets

Under mean-model misspecification, regression coefficients need not be interpreted as local derivatives of $\mathbb{E}[Y]$1. A different interpretation views them as linear summaries of how much the population mean of $\mathbb{E}[Y]$2 can be changed by changing the distribution of $\mathbb{E}[Y]$3. In the univariate case,

$\mathbb{E}[Y]$4

so the regression slope is the linear mean impact of $\mathbb{E}[Y]$5 on $\mathbb{E}[Y]$6 divided by the linear mean impact of $\mathbb{E}[Y]$7 on itself. In multiple regression, the coefficient $\mathbb{E}[Y]$8 becomes the linear partial mean impact slope, defined through distributional disturbances that leave the means of the other covariates unchanged (Brannath et al., 2014).

Robustification can preserve the mean-regression objective, but it may also alter the population target. In high-dimensional heteroscedastic mean regression with pseudo-Huber loss,

$\mathbb{E}[Y]$9

the robustified target

$m(x)=\mathbb{E}[Y\mid X=x].$0

coincides with the true mean parameter $m(x)=\mathbb{E}[Y\mid X=x].$1 in the particular case of a symmetric conditional distribution of $m(x)=\mathbb{E}[Y\mid X=x].$2 given $m(x)=\mathbb{E}[Y\mid X=x].$3, but differs from $m(x)=\mathbb{E}[Y\mid X=x].$4 in general. The paper therefore distinguishes the support of $m(x)=\mathbb{E}[Y\mid X=x].$5 from the support of $m(x)=\mathbb{E}[Y\mid X=x].$6, and shows that these supports may differ substantially even for small $m(x)=\mathbb{E}[Y\mid X=x].$7 (Hermann et al., 2020).

Heavy-tailed regression theory sharpens this point. Robust mean estimators such as median-of-means, Catoni’s estimator, and trimmed means recover sub-Gaussian-type deviation bounds under finite variance, and the same design principle extends to regression through uniform median-of-means procedures, distance oracles, and tournament methods. In this literature, regression mean estimation is treated explicitly as a robust mean-estimation problem for squared losses, replacing empirical averages and ERM by robust blockwise comparisons that remain valid under heavy tails (Lugosi et al., 2019).

4. Tail-conditioned and specialized mean functionals

Some regressions redefine the target mean itself. Regressions under adverse conditions model

$m(x)=\mathbb{E}[Y\mid X=x].$8

with

$m(x)=\mathbb{E}[Y\mid X=x].$9

Estimation proceeds in two steps: first a quantile regression for $m(x;\beta)=x'\beta,$0, then a tail-restricted squared-error regression

$m(x;\beta)=x'\beta,$1

In that setting, RegMean is explicitly the conditional mean of $m(x;\beta)=x'\beta,$2 under the adverse event that the distress variable $m(x;\beta)=x'\beta,$3 exceeds its conditional quantile, and the paper interprets this as a regression for Marginal Expected Shortfall (Dimitriadis et al., 2023).

Biased mean regression shifts the target from $m(x;\beta)=x'\beta,$4 to

$m(x;\beta)=x'\beta,$5

where $m(x;\beta)=x'\beta,$6 is a user-chosen margin. The method minimizes the superexpectation error

$m(x;\beta)=x'\beta,$7

and is equivalent to minimizing the associated deviation under the constraint $m(x;\beta)=x'\beta,$8. The paper proves two equivalence results: biased mean regression is equivalent to quantile regression for an appropriate parameterization and is equivalent to ordinary least squares when $m(x;\beta)=x'\beta,$9 (Malandii et al., 27 Mar 2026).

In survival analysis, Bayesian nonparametric mean residual life regression treats

$$\beta=\arg\min_b \mathbb{E}\big[(Y-X^\top b)^2\big].$$0

as the primary regression functional. Dirichlet process mixture modeling of the joint law of $$\beta=\arg\min_b \mathbb{E}\big[(Y-X^\top b)^2\big].$$1 implies a mixture representation

$$\beta=\arg\min_b \mathbb{E}\big[(Y-X^\top b)^2\big].$$2

with time- and covariate-dependent weights. The framework is extended to multiple groups through a dependent Dirichlet process prior with common atoms and group-specific weights (Poynor et al., 2014).

5. Uncertainty-aware, online, and distribution-sensitive RegMean

A prominent modern theme is the joint modeling of a regression mean and a dispersion process. In nonparametric mean–variance regression,

$$\beta=\arg\min_b \mathbb{E}\big[(Y-X^\top b)^2\big].$$3

the regression mean $$\beta=\arg\min_b \mathbb{E}\big[(Y-X^\top b)^2\big].$$4 competes with the precision $$\beta=\arg\min_b \mathbb{E}\big[(Y-X^\top b)^2\big].$$5 for explanatory power. The paper identifies a signal–to–noise ambiguity in overparameterized models and reports a sharp phase transition between underfitting and collapse, driven by regularization. The stable regime $$\beta=\arg\min_b \mathbb{E}\big[(Y-X^\top b)^2\big].$$6 is crossed by the minor diagonal $$\beta=\arg\min_b \mathbb{E}\big[(Y-X^\top b)^2\big].$$7, which reduces hyperparameter search from two dimensions to one. In that regime, the regression mean captures the main structure while the variance captures heteroskedastic residual variation without collapsing (Wong-Toi et al., 27 Nov 2025).

Streaming kernel regression studies the same object in sequential form. With a kernel $$\beta=\arg\min_b \mathbb{E}\big[(Y-X^\top b)^2\big].$$8, Gram matrix $$\beta=\arg\min_b \mathbb{E}\big[(Y-X^\top b)^2\big].$$9, and adaptive regularization $x'\beta$0, the online regression mean is

$x'\beta$1

The paper allows $x'\beta$2 to be a predictable function of past data, derives uniform confidence bounds for $x'\beta$3, constructs online upper and lower confidence bounds for the unknown noise variance, and sets

$x'\beta$4

This makes the regression mean, variance estimate, and regularization parameter all adaptive in the same streaming procedure (Durand et al., 2017).

For count data, mean-parameterized Conway–Maxwell–Poisson regression makes the mean itself the primary parameter: $x'\beta$5 The rate parameter $x'\beta$6 is defined implicitly so that the CMP mean equals $x'\beta$7, which lets the model use a log-linear mean specification directly. The paper emphasizes that the mean and dispersion are orthogonal, Poisson regression is recovered at $x'\beta$8, and the resulting MATLAB routine is up to an order of magnitude faster than the current software to fit standard CMP models and over two orders of magnitude faster than the recently proposed hyper-Poisson model (Huang, 2016).

A nearby contrast is distribution regression, which treats mean regression as the Gaussian special case of a full error-density likelihood. If the error density is Gaussian, maximizing the log-likelihood is equivalent to OLS; if it is Laplace, it corresponds to median regression; and the proposed nonparametric likelihood estimates the full error density rather than fixing a quadratic loss in advance. In that sense, ordinary RegMean appears as one location-functional special case within a broader distributional framework (Chen et al., 2017).

6. RegMean as a model-merging algorithm

In a distinct usage, RegMean denotes a training-free model-merging method for neural networks. Given task-specific candidate models $x'\beta$9 sharing the same architecture, RegMean merges each linear layer by solving a layerwise regression problem that matches the merged layer’s output to the candidate layers’ outputs on observed features. For a linear layer $L^2$0, with candidate weights $L^2$1 and feature Gram matrices $L^2$2, the merged weight is obtained in closed form as

$L^2$3

Other parameters are merged by simple averaging. This yields explainability and computational efficiency, but it merges each linear layer independently and therefore overlooks how earlier layers propagate features through the merged network (Nguyen et al., 5 Aug 2025).

RegMean++ keeps the same closed-form regression structure but replaces candidate-model activations by merge-model activations $L^2$4, so the Gram matrices reflect the feature space actually induced by the merged model. The stated motivation is to incorporate intra- and cross-layer dependencies while retaining the same analytic solution form. On the 8-task CLIP benchmark, the reported average accuracy improves from 82.4 to 84.4 for ViT-B/32, from 86.0 to 87.2 for ViT-B/16, and from 90.4 to 91.0 for ViT-L/14. The same study reports improvements in in-domain and out-of-domain generalization, sequential merging, robustness under several types of distribution shifts, and large-scale settings up to 20 tasks (Nguyen et al., 5 Aug 2025).

This architectural usage is conceptually separate from the statistical uses of RegMean, but the shared formal idea is unmistakable: in both cases, the mean object is obtained by solving a regression problem against an explicitly chosen reference behavior. In classical statistics that reference is a conditional expectation, a tail-conditioned expectation, or a mean residual life; in model merging it is the average layerwise behavior of candidate models.