Constrained Weighted Least Squares Estimator

• CWLS is a generalization of weighted least squares that incorporates linear or convex constraints to encode domain knowledge like monotonicity, positivity, and sparsity.
• It is typically solved using quadratic programming and iteratively reweighted least squares, making it widely applicable in GLMs, compositional regression, and adaptive filtering.
• The estimator balances bias and variance by reducing variance through constraints while potentially introducing bias when true values approach or cross constraint boundaries.

A Constrained Weighted Least Squares (CWLS) estimator constitutes a generalization of classical weighted least squares, incorporating linear or convex side constraints directly into the estimation procedure. The CWLS formulation is central to many contemporary statistical, econometric, and signal processing contexts, including generalized linear models, compositional regression, shape-restricted nonparametric estimation, and adaptive filtering. The addition of constraints to the Euclidean loss equip CWLS with the ability to encode subject-matter knowledge (monotonicity, positivity, sparsity) and enhance regularity under ill-conditioned or high-dimensional regimes.

1. Formal Definition and Optimization Problem

CWLS extends the standard weighted least squares criterion by imposing linear or convex constraints. Given observations $y \in \mathbb{R}^n$, predictors $X \in \mathbb{R}^{n \times p}$, and a positive-definite weight matrix $W \in \mathbb{R}^{n \times n}$, the generic CWLS problem is

$$\min_{\beta \in \mathbb{R}^p} \; (y - X\beta)^\top W (y - X\beta) \quad \text{s.t.} \quad C\beta = d$$

or, more generally, subject to $A\beta \leq b$, $C\beta = d$ for linear inequality and equality constraints (Lang et al., 2017). This paradigm appears in generalized linear models (GLMs) as the quadratic approximation in an iteratively-reweighted least squares (IRLS) algorithm, but now restricted to a feasible set defined by the constraints (Masselot et al., 22 Sep 2025).

For models with exponential family noise or nonconstant variance, weights can encode inverse variance, e.g., in GLMs $$W_{ii} = [g'(\mu_i)]^2/\operatorname{Var}(y_i|\mu_i)$$, where $\mu = g^{-1}(X\beta)$ and $g(\cdot)$ is the link function (Masselot et al., 22 Sep 2025). In kernel smoothing contexts, weights capture local proximity via kernels and bandwidth parameters (Yagi et al., 2016).

2. Algorithmic Solution Methods

In most practical applications, the CWLS problem is solved via quadratic programming (QP). The constraints transform the unconstrained weighted least squares normal equations to a convex QP, which admits efficient solution via interior-point, active-set, or dual algorithms (Goldfarb–Idnani) (Masselot et al., 22 Sep 2025, Tsagris, 17 Nov 2025).

In the GLM setting, the algorithm proceeds as:

1. Compute working response $y^* = \eta + (y - \mu) g'(\mu)$, where $\eta = X\beta$.
2. Update weights $W_{ii}$ according to model variance and link derivative.
3. Solve constrained QP:

$$\min_\beta (y^* - X\beta)^\top W (y^* - X\beta) \quad \text{s.t.} \quad A\beta \leq b$$

The update for $\beta$ is:

$$\min_\beta \frac{1}{2}\beta^\top Q \beta + c^\top\beta \quad \text{s.t.} \quad A\beta \leq b$$

where $Q = X^\top W X$ and $c = -X^\top W y^*$ (Masselot et al., 22 Sep 2025).

In compositional regression (simplicial regression), constraints enforce that the coefficient vector lies in the probability simplex (all entries nonnegative, sum to one), and the QP includes both equality (sum constraint) and box (componentwise) constraints (Tsagris, 17 Nov 2025).

For kernel-based nonparametric regression under monotonicity and convexity/concavity constraints, SCKLS uses a block-diagonal QP with a quickly growing number of constraints—adjacent-grid concavity constraints enable scalable computation via cutting-plane methods (Yagi et al., 2016).

3. Statistical Properties and Inference

Covariance and Bias

The covariance matrix of the CWLS estimator is generally smaller than the unconstrained analogue, due to restriction of estimation space (Lang et al., 2017):

$$\operatorname{Cov}(\hat\beta_{CWLS}) = (X^\top W X)^{-1} - (X^\top W X)^{-1} C^\top [C (X^\top W X)^{-1} C^\top]^{-1} C (X^\top W X)^{-1}$$

Imposing constraints generally introduces bias if the true $\beta^*$ pushes up against or lies outside the feasible region, but concurrently decreases variance (Masselot et al., 22 Sep 2025). The net impact on root mean squared error (RMSE) is context dependent; when the true coefficients are near the feasible boundary, the variance reduction can more than offset the induced bias.

Degrees of Freedom

Constraints reduce the effective number of free parameters. For linear constraints in GLMs, the observed degrees of freedom is $p - m_a$ with $m_a$ the number of active constraints. The expected degrees of freedom (EDF) can be quantified as

$$\mathrm{edf} = \mathbb{E}[p - m_a] = p - \sum_{k=0}^m k P(m_a = k)$$

where $P(m_a = k)$ is estimated via Monte Carlo from the unconstrained normal distribution (Masselot et al., 22 Sep 2025).

Inference Techniques

Asymptotic inference for CWLS estimators employs the truncated multivariate normal (TMVN) law:

$$C\beta \sim \mathrm{TMVN}(C\beta^*,\,C\Sigma^* C^\top,\;\ell,\;u)$$

Practical inference involves simulating TMVN draws, solving $C \beta^{(j)} = z^{(j)}$ for each, and producing empirical confidence intervals (Masselot et al., 22 Sep 2025). This respects the constraints and corrects for truncation-induced bias.

4. Applications in Statistical Modeling

CWLS supports model fitting in high-dimensional, collinear, or structurally constrained scenarios. Key domains include:

• Generalized Linear Models (GLMs) with constraints: CIRLS implements CWLS via iterative QPs, accommodating side knowledge (positivity, monotonicity, etc.) and robustifying ill-conditioned problems (Masselot et al., 22 Sep 2025).
• Compositional Data Analysis: In transformation-free simplicial–simplicial regression, CWLS enforces row-stochastic constraints and enables efficient estimation via CIRLS, outperforming EM algorithms in scalability (Tsagris, 17 Nov 2025).
• Shape-Constrained Nonparametrics: SCKLS utilizes kernel-weighted loss with monotonicity/convexity constraints, yielding estimators that respect economic or scientific theory while maintaining optimal convergence rates (Yagi et al., 2016).
• Autocovariance/Spectral Density Estimation: Weighted shape-constrained CWLS projects empirical autocovariances onto valid Markov structure spaces using Fourier-domain weights for optimal consistency and efficiency (Song et al., 6 Aug 2024).
• Adaptive Filtering: Relaxed constrained LS (rCLS) employs a penalty parameter $\lambda$ to interpolate between unconstrained RLS and strict constraints; its bias, covariance, and mean-square error are analytically tractable (Arablouei et al., 2014).

5. Computational Complexity and Convergence

CWLS estimation generally requires solving p-dimensional QPs with linear constraints. In CIRLS for GLMs, the dominant cost per iteration is $O(np^2)$ for weight and matrix computations plus $O(p^3)$ for solving the QP (Tsagris, 17 Nov 2025). In nonparametric contexts (e.g., SCKLS), the number of constraints can grow as $O(m^2)$ for $m$ grid points, mitigated by constraint-reduction and cutting-plane methods (Yagi et al., 2016).

Convergence properties depend on the convexity of the subproblem and the monotonicity of the objective. CIRLS exhibits guaranteed convergence to a unique global optimum under strict convexity, regular QP solvers, and monotonic increase in the log-likelihood (Tsagris, 17 Nov 2025, Masselot et al., 22 Sep 2025, Arablouei et al., 2014). In adaptive filtering (rCLS), both mean and mean-square stability hold under suitable step-size choices, with geometric convergence in mean (Arablouei et al., 2014).

6. Bias–Variance Trade-Off and Practical Guidance

Empirical and theoretical analyses reveal that constraints typically increase bias when the true parameter vector lies outside the feasible region or near the boundary, while always reducing estimator variance by removing unstable directions from coefficient space (Masselot et al., 22 Sep 2025, Arablouei et al., 2014). Optimal RMSE is achieved when tight, correct constraints lead to the greatest variance reduction with minimal bias.

Key practical recommendations include:

• Set constraints based on robust subject-matter knowledge.
• Check feasibility against unconstrained estimates to avoid overly stringent restrictions.
• Use EDF to calibrate constraint impact; excessively low EDF signals over-constraining.
• Conduct simulation or bootstrap evaluation of bias–variance behavior under proposed constraints (Masselot et al., 22 Sep 2025).

7. Connections to Related Estimation Paradigms

CWLS generalizes both the classical least squares and generalized least squares frameworks, applying to constrained best linear unbiased estimation (CBLUE), shape-restricted regression, and convex-penalized estimation procedures (Lang et al., 2017, Yagi et al., 2016, Song et al., 6 Aug 2024). As penalty parameters $\lambda \to \infty$, the solution approaches hard-constrained LS (CLS); as $\lambda \to 0$, it recovers unconstrained RLS (Arablouei et al., 2014). In all cases, explicit solution forms and analytic variance expressions are available when the constraint and design matrices have full rank.

CWLS acts as a versatile and tractable mechanism for embedding structural domain knowledge and achieving robust, interpretable estimation in diverse statistical and engineering settings.