Gauss-Optimal Linear Combination

• Gauss-optimal linear combination is a method for constructing linear estimators that minimize error by optimally weighting data under Gaussian or near-Gaussian assumptions.
• It leverages advanced techniques—such as Fourier analysis, orthogonal polynomials, and semidefinite programming—to achieve accuracy improvements like O(1/n) rates over classical methods.
• Applications include enhanced Berry–Esseen bounds, optimal quadrature for exact moment matching, and risk-minimizing linear estimation in high-dimensional or mixture model settings.

A Gauss-optimal linear combination refers to an explicit or constructively obtained linear functional or estimator that minimizes a prescribed risk or error measure, typically under Gaussian or nearly Gaussian statistical assumptions. In both the analysis of random variable summation and statistical estimation or quadrature, there exist universal constructions yielding rates or levels of accuracy associated with Gaussian optimality that surpass traditional, non-adaptive linear forms.

1. Foundations in Probability and Summation: Beyond the Classical Berry–Esseen

The Berry–Esseen theorem establishes bounds for the Gaussian approximation of normalized sums of independent, identically distributed (i.i.d.) random variables. For variables $X_1,\dots,X_n$ with mean zero, variance $\sigma^2$, and third absolute moment $\rho$, the classical rate for the maximal deviation from the standard normal CDF is $O(1/\sqrt n)$: $$\sup_{x\in\mathbb R}\Bigl|\Pr\bigl(S_n/(\sigma\sqrt n)\le x\bigr)-\Phi(x)\Bigr| \leq C \frac{\rho}{\sigma^3\sqrt n}.$$ However, under the stronger assumption of finite fourth moment $\delta^4$, Klartag–Sodin show that by considering linear combinations

$$L = \sum_{i=1}^n \theta_i X_i, \quad \|\theta\|_2=1,$$

it is possible to achieve a rate $O(1/n)$, not attainable by the unweighted sum (Klartag et al., 2010). For i.i.d. $X_i$ with these moment bounds, both random and explicit deterministic choices of coefficients $\theta$, such as the 4-cycle pattern

$$\theta^* = \frac{1}{\sqrt{3n/2}}\big(1,\,1/\sqrt2,\,-1,\,-1/\sqrt2,\ldots\big),\quad n \text{ divisible by }4,$$

yield

$$\sup_{x}\Bigl|\Pr(L\le x)-\Phi(x)\Bigr| \leq C\,\frac{\delta^4}{n}.$$

For non-identically distributed $X_i$ with unit variance and average fourth moment $\bar\delta^4$, most $\theta\in S^{n-1}$ (excluding at most a fraction $p$ of the sphere) satisfy

$$\sup_{x} \Bigl|\Pr(L\le x)-\Phi(x)\Bigr| \leq C(p) \frac{\bar\delta^4}{n}.$$

The proof employs Fourier-analytic smoothing and geometric properties of the high-dimensional sphere, with explicit constructions built using Diophantine approximation methods (Klartag et al., 2010). This phenomenon demonstrates that one can obtain Berry–Esseen-beating accuracy in Gaussian approximation via appropriate linear combinations.

2. Gauss-Optimality in Quadrature: Exactness and Moment Matching

Gauss quadrature rules, in the context of linear functionals $\mathcal{L}$ on polynomials, yield unique interpolatory formulas that match the first $2n$ moments using only $n$ nodes. For any linear functional with known moments $m_k = \mathcal{L}[x^k]$, the existence of formal orthogonal polynomials (FOPs) leads to the construction of nodes and weights $(x_i, \omega_i)$ such that

$Q_n[f] = \sum_{i=1}^n \omega_i f(x_i)$

is exact for all polynomials of degree $\leq 2n-1$. The matching moment property states: $$Q_n[x^k] = m_k = \mathcal{L}[x^k], \qquad k=0,1,\dots,2n-1.$$ This makes the Gauss quadrature rule uniquely optimal: it minimizes the number of nodes required for this degree of exactness. The nodes are the roots of the degree-$n$ FOP; weights are obtained from the orthogonal structure and explicit formulas involving Hankel determinants or the Christoffel formula (Pozza et al., 2019). In limiting or degenerate settings (incurable breakdown), the rule can be exact for all polynomials, equivalent to exact eigenvalue computation in Krylov or Lanczos methods.

3. Gauss-Optimal Linear Estimation in Statistical Inference

In high-dimensional inference, the goal is often to find a linear combination or transformation that minimizes a given risk. When the statistical model is linear, with Gaussian noise and ellitope-constrained signal set,

$\omega = A x + \sigma \xi,$

the Gauss-optimal linear estimator $H_*$ solves

$$\min_H \sup_{x\in X} E\,\|B x - H(Ax + \sigma\xi)\|_2^2,$$

with $X$ an intersection of quadratic constraints (ellitope) (Juditsky et al., 2016). The risk splits as

$$R^2(H) = \sigma^2 \operatorname{Tr}(H H^T) + \max_{Q \in \mathcal{Q}}\operatorname{Tr}((B-HA)^T (B-HA) Q),$$

where $\mathcal{Q}$ is the convex hull of allowable signal covariances.

The minimization is equivalent to a semidefinite program (SDP), with explicit block-matrix form constraints, and can be solved efficiently. Special cases include identity transform and covariance shrinkage, yielding classical ridge formulas. The main result is that such $H_*$ achieves risk within a logarithmic factor of the minimax risk: $$\sup_{x\in X} E\|H_*(A x + \sigma \xi) - B x\|_2 \leq O\left(\sqrt{\ln \left( \frac{K M_*^2}{[X]^2} \right)}\right) [X],$$ with $K$ the number of quadratic constraints, and $[X]$ the minimax risk.

4. Bayes-Optimal Linear Combination of Data-Driven Estimators

In estimation from generalized linear models with Gaussian measurement matrices, key recent work (Mondelli et al., 2020) establishes that the Bayes-optimal combination of two estimators—one linear $\hat{x}^L$ (data-weighted sum of features), and one spectral $\hat{x}^s$ (principal eigenvector of a data-dependent matrix)—is itself linear when the underlying signal is Gaussian: $$\hat{x}^{\mathrm{opt}} = \theta_* \hat{x}^L + \hat{x}^s,$$ where the optimal scalar $\theta_*$ is computed via

$$\theta_* = \frac{\rho_L - \rho_s q}{\rho_s - \rho_L q}.$$

Here $\rho_L, \rho_s, q$ represent normalized correlations and cross-correlations between estimators and signal, determined by model quantities and preprocessing functions. This form is rigorously derived using high-dimensional limit theory and Approximate Message Passing (AMP); the combined estimator provably yields maximal overlap with the true signal under the Gaussian prior.

5. Gauss-Optimal Linear Design under Non-Gaussian and Mixture Models

In problems where both the signal $x$ and the noise $n$ are mixtures of Gaussians (GM), the design of a linear transformation $H$ to minimize MMSE in models of the form $y = Hx + n$ becomes nontrivial (Flåm et al., 2012). While the optimal MMSE estimator is generally nonlinear and the MMSE is not convex in $H$, practical optimization proceeds using stochastic gradient algorithms. These algorithms leverage explicit formulas for the gradient of the MMSE with respect to $H$, grounded in the structure of the mixture posteriors and their derivatives.

Notably, in the pure Gaussian case, MMSE is convex in $H$ with a unique global minimum, but for general mixtures, the optimization landscape is nonconvex, exhibiting multiple local minima and maxima. This has practical implications for the design of channel precoders and pilot matrices in signal processing; for example, an increase in pilot power can degrade MMSE in certain regimes by exacerbating mixture ambiguities, a phenomenon not present in the Gaussian setup (Flåm et al., 2012).

6. Mechanisms of Improvement: High-dimensional Geometry, Orthogonality, and Convexity

The enhancement from Berry–Esseen $O(1/\sqrt n)$ to Gauss-optimal $O(1/n)$ arises from harmonizing the linear weights with the high-dimensional geometry of the sphere $S^{n-1}$, ensuring cancellation of higher-order cumulants (Klartag et al., 2010). In quadrature, optimality is certified by orthogonality properties of polynomials relative to the chosen moment sequence; in estimation, by maximizing worst-case or average-case risk via explicit matrix inequalities or functional optimization. When statistical noise is Gaussian, the optimal linear structure frequently manifests as shrinkage or ridge regularization. Extensions and open questions include: removal of logarithmic factors in rates, explicit deterministic constructions for all regimes, and optimal weights under heavy-tailed or sub-Gaussian settings (Klartag et al., 2010).

---

This summary incorporates the essential phenomena and explicit methods underlying Gauss-optimal linear combinations as developed in the modern literature, grounding each advancement in technical results, algorithmic procedures, and illustrating their roles in summation, quadrature, estimation, and high-dimensional statistics.