Mean Square Optimal Estimation

• Mean square optimal estimation is a paradigm that minimizes the mean squared error by deriving the MMSE estimator, defined as the conditional mean.
• Key results show that linearity holds under a precise matching condition between source and noise, with the Gaussian case uniquely permitting multi-SNR linearity.
• Extensions to vector cases require coordinate-wise transformations and independence conditions, ensuring optimal linear estimation across dimensions.

Mean square optimal estimation is a central paradigm in statistical inference, signal processing, and information theory, in which one seeks an estimator that minimizes the expected squared error (mean squared error, MSE) between an unknown parameter or signal and its estimate, typically under a physical observation or system model with noise or other uncertainty. The classical minimum mean square error (MMSE) estimator is the conditional mean. However, the precise properties, structure, and (especially) linearity of the mean square optimal estimator depend intricately on the joint distribution of the signal and noise. The question of when the optimal estimator is linear—and, more generally, the conditions and consequences of MSE-optimality—has profound implications for theory and practice.

1. Conditions for Linearity of the Mean Square Optimal Estimator

Let $X$ (source) and $Z$ (noise) be independent random variables, with $Y = X + Z$ observed. The MMSE estimator is $h^*(Y) = E[X | Y]$. It is well known that if $X, Z$ are jointly Gaussian, the MMSE estimator is linear:

$$h^*(Y) = \frac{\gamma}{\gamma + 1} Y, \quad \gamma = \frac{\sigma_X^2}{\sigma_Z^2}.$$

The general question—when is $h^*(Y)$ linear?—is addressed by deriving a necessary and sufficient condition using the characteristic functions $F_X(\omega)$ and $F_Z(\omega)$. For $L_p$ distortion ($\Phi(x)=|x|^p$ with even $p$), Theorem 1 shows optimal linearity if and only if the following differential equation is satisfied:

$$\sum_{m=0}^{p-1} \binom{p-1}{m} F_X^{(m)}(\omega) F_Z^{(p-1-m)}(\omega) \left(\frac{k-1}{k}\right)^m = 0$$

for the linear estimator $h(Y)=k Y$.

For mean square error ($p=2$), this reduces to the "matching condition":

$$F_X(\omega) = [F_Z(\omega)]^{\gamma}, \qquad \gamma = \frac{\sigma_X^2}{\sigma_Z^2}$$

i.e., the source characteristic function must be a positive power (possibly fractional) of the noise characteristic function. This is both necessary and sufficient for the linearity of $h^*(Y)$ in general.

2. Existence and Uniqueness of Matching Distributions

The matching condition has several profound consequences:

• If $\gamma$ is a natural number, any $F_Z(\omega)$ yields a valid matching $F_X(\omega)= [F_Z(\omega)]^{\gamma}$: $X$ is a sum of $\gamma$ independent copies of $Z$.
• More generally, $F_Z(\omega)^{\gamma}$ is a characteristic function if and only if $Z$ is infinitely divisible (e.g., Gaussian, Poisson, stable law).
• Uniqueness: If $F_Z$ is analytic (i.e., the distribution has moments of all orders), the matching $F_X$ is unique and determined by the moments.

Parameter	Matching Existence	Matching Uniqueness
$\gamma\in\mathbb{N}$	Always	By construction
$Z$ infinitely divisible	For all $\gamma>0$	If $F_Z$ analytic
$\sigma_X^2 = \sigma_Z^2$	$F_X=F_Z$ (identical distributions)	Unique

If $X$ and $Z$ have equal variances ($\gamma = 1$), the only way $h^*(Y)$ is linear is if $X$ and $Z$ are identically distributed, i.e., $f_X(x)=f_Z(x) \; \forall x$.

3. The Uniqueness of the Gaussian Source–Noise Pair

A key result (Theorem 5) is that the Gaussian source–channel pair is uniquely characterized by the property that linearity of $h^*(Y)$ holds for more than one signal-to-noise ratio (SNR) value. If, for two different SNRs $\gamma_1$, $\gamma_2$, the matching condition can be satisfied—$$F_X(\omega) = F_Z(\omega)^{\gamma_1} = F_Z(\alpha\omega)^{\gamma_2}$$—then log $F_Z(\omega)$ must be quadratic, so $Z$ is Gaussian.

Thus: For any non-Gaussian pair, the linear estimator can be optimal at most for a single SNR. For all SNRs, only the jointly Gaussian case yields linear $h^*(Y)$.

4. Asymptotic Linearity at Low and High SNR

For general (not necessarily matching) source–noise pairs:

• As $\gamma \to 0$ (low SNR, i.e. the noise dominates), if $Z$ is Gaussian, $h^*(Y)$ becomes asymptotically linear for any source.
• As $\gamma \to \infty$ (high SNR), if $X$ is Gaussian, $h^*(Y)$ becomes asymptotically linear for any noise.

This "asymptotic robustness" explains the empirical success of linear estimators such as the Wiener filter in diverse regimes, even when the exact matching condition fails.

5. Vector Case: Transformation and Coordinate-wise Matching

In the vector observation setting ($Y = X + Z$ with $X$, $Z$ independent vectors), optimal linearity is more restrictive. The necessary and sufficient condition is that, after a linear transformation $U$ (which diagonalizes $R_X R_Z^{-1}$), the components of $UX$ and $UZ$ satisfy

$$\frac{\partial}{\partial \omega_i} \log F_{UX}(\omega) = \lambda_i \frac{\partial}{\partial \omega_i} \log F_{UZ}(\omega)$$

for each $i$, where $\lambda_i$ are the eigenvalues of $R_X R_Z^{-1}$.

Moreover, the transformed sources and noise must satisfy certain independence or conditional independence conditions: in the case of distinct eigenvalues, optimality of a linear estimator requires independence of those coordinates (no spurious dependencies across dimensions).

6. Consequences and Broader Implications

• The mean square optimal (MMSE) estimator is linear if and only if the source and noise distributions satisfy the precise matching condition $F_X(\omega)= [F_Z(\omega)]^{\gamma}$.
• The only source–noise pair for which MMSE linearity persists for all SNRs is the Gaussian–Gaussian pair.
• Linear estimators are asymptotically optimal at extreme SNRs, provided either the source or noise is Gaussian.
• In practical estimation, the use of linear estimators is justified either when the matching condition holds or when operating in extreme SNR regimes.
• In the vector case, these results extend but require block-wise (coordinate-wise) matching—after appropriate diagonalization—and further require conditional independence conditions not needed in the scalar case.

7. Summary Table of Linearity Conditions

Scenario	Matching/Optimality Condition	Estimator
Scalar MSE ($Y = X + Z$)	$F_X(\omega) = [F_Z(\omega)]^\gamma$	$h^*(Y) = kY$
Equal variances ($\gamma=1$)	$F_X(\omega) = F_Z(\omega)$	$h^*(Y) = \frac{1}{2}Y$
Multiple SNR linearity	Gaussian only	$h^*(Y) = k(\gamma) Y$
Low SNR, Gaussian noise	Asymptotically linear, any source	$h^*(Y) \approx kY$
High SNR, Gaussian source	Asymptotically linear, any noise	$h^*(Y) \approx kY$
Vector case	$$\partial_{\omega_i}\log F_{UX} = \lambda_i \partial_{\omega_i}\log F_{UZ}$$ for each $i$; coordinate-wise independence required	Linear in $UY$

These results precisely characterize the conditions under which mean square optimal estimation is linear and establish the Gaussian case as uniquely linear at all SNRs, situating the Wiener filter and related techniques on a rigorous foundation (Akyol et al., 2011).

---

References (by arXiv id):

• (Akyol et al., 2011) On Conditions for Linearity of Optimal Estimation