Regularized SAA: Methods & Insights

• Regularized SAA is a stochastic optimization method that augments classical SAA with explicit regularization to balance empirical fit and promote structural properties like low-rankness and robustness.
• It employs techniques such as low-rank inducing penalties, scenario-based adjustments, and adaptive DRO integration to achieve finite-sample guarantees and computational efficiency in high-dimensional settings.
• The RSAA framework offers practical advantages across applications like portfolio optimization and multi-product newsvendor problems by significantly reducing sample complexity compared to traditional approaches.

Regularized Sample Average Approximation (Regularized SAA, or RSAA) comprises a class of stochastic optimization methods that enhance the classical sample average approximation (SAA) framework by incorporating explicit regularization. The primary goals of RSAA methods are to improve statistical efficiency, induce desirable solution structure (such as low-rankness or robustness), and balance the trade-off between empirical fit and generalization. These methods have seen significant advancement in three core streams: (i) regularization for high-dimensional or structured estimation, (ii) scenario-based regularization for targeted robustification, and (iii) adaptive hybridization of SAA with distributionally robust optimization (DRO). Recent works have established foundational theory, finite-sample guarantees, algorithmic reformulations, and broad practical advantages across these paradigms (Liu et al., 2019, Fonseca et al., 20 Nov 2025, Jin et al., 26 Aug 2025).

1. Theoretical Foundations of Regularized SAA

The canonical stochastic program minimizes an expected loss $\mathcal{F}(x) = \mathbb{E}_\xi[f(x,\xi)]$ over feasible $x \in \mathcal{X}$, where $\xi$ is a random parameter. SAA approximates this by minimizing the empirical mean: $$\min_{x \in \mathcal{X}}~ \frac{1}{n}\sum_{i=1}^n f(x, \xi_i).$$ Regularized SAA augments this objective with a penalty term $R(x)$, producing the general form: $$\min_{x\in\mathcal{X}}~ \frac{1}{n}\sum_{i=1}^n f(x, \xi_i) + \lambda R(x),$$ where $\lambda \geq 0$ calibrates the regularization magnitude. Distinct choices of $R(x)$ and $\lambda$ yield targeted regularized SAA paradigms for specific statistical and computational goals.

2. Low-Rank Regularization in High-Dimensional SAA

For matrix-valued optimization problems—especially over the positive semidefinite cone $$S_p = \{X \in \mathbb{R}^{p\times p}: X \succeq 0\}$$—classical SAA requires $x \in \mathcal{X}$0 samples to bound suboptimality by $x \in \mathcal{X}$1, leading to prohibitive sample complexity in large dimensions (Liu et al., 2019). RSAA introduces low-rank inducing penalties:

• Nuclear norm penalty: $x \in \mathcal{X}$2.
• Folded-concave penalty (MCP): $x \in \mathcal{X}$3, $x \in \mathcal{X}$4.

Under assumptions of low-rankness (rank $x \in \mathcal{X}$5 for $x \in \mathcal{X}$6), spectral smoothness, and sub-exponential loss tails, RSAA achieves: $x \in \mathcal{X}$7 where $x \in \mathcal{X}$8 is in-sample suboptimality. The resulting sample complexity scales as $x \in \mathcal{X}$9—nearly linear in $\xi$0, breaking the classical $\xi$1 barrier without requiring restricted strong convexity. Theoretical guarantees hold for stationary points satisfying the significant subspace second-order necessary condition (S³ONC), attainable by local solvers with suitable warm starts (Liu et al., 2019).

3. Scenario-Based Regularization for Robustness

Scenario-based regularized SAA (SBR-SAA) augments the SAA with a regularizer penalizing the (sub)gradient norm of the loss at a user-specified set of $\xi$2 "regularization" scenarios $\xi$3: $\xi$4 with weights $\xi$5, $\xi$6 (Fonseca et al., 20 Nov 2025). The SBR-SAA objective is then

$\xi$7

where $\xi$8 governs the robustness-accuracy trade-off.

Key theoretical properties include:

• Equivalence to decision-dependent WDRO: SBR-SAA solves a distributionally robust problem where the radius of the Wasserstein ball is allowed to depend on $\xi$9 via $$\min_{x \in \mathcal{X}}~ \frac{1}{n}\sum_{i=1}^n f(x, \xi_i).$$0.
• Finite-sample guarantees: With radius parameter $$\min_{x \in \mathcal{X}}~ \frac{1}{n}\sum_{i=1}^n f(x, \xi_i).$$1 (for constants $$\min_{x \in \mathcal{X}}~ \frac{1}{n}\sum_{i=1}^n f(x, \xi_i).$$2), SBR-SAA achieves high-probability out-of-sample bounds.
• Asymptotic consistency: As $$\min_{x \in \mathcal{X}}~ \frac{1}{n}\sum_{i=1}^n f(x, \xi_i).$$3, the sequence of SBR-SAA optimal values converges to the true expected loss minimizer with probability 1.

Scenario selection can target statistical diversity (e.g., via $$\min_{x \in \mathcal{X}}~ \frac{1}{n}\sum_{i=1}^n f(x, \xi_i).$$4-medoids or scenario reduction algorithms) or problem-specific adverse events for tailored robustness. Computationally, SBR-SAA transforms the WDRO problem into a more tractable mixed-integer second-order conic program (MISOCP) or SOCP.

4. Harmonizing Optimization: Adaptive Convex Combination of SAA and DRO

Harmonizing Optimization (HO) generalizes RSAA by convexly combining the SAA and a moment-based DRO objective. Let $$\min_{x \in \mathcal{X}}~ \frac{1}{n}\sum_{i=1}^n f(x, \xi_i).$$5,

$$\min_{x \in \mathcal{X}}~ \frac{1}{n}\sum_{i=1}^n f(x, \xi_i).$$6

where $$\min_{x \in \mathcal{X}}~ \frac{1}{n}\sum_{i=1}^n f(x, \xi_i).$$7 is the empirical distribution and $$\min_{x \in \mathcal{X}}~ \frac{1}{n}\sum_{i=1}^n f(x, \xi_i).$$8 is a (typically moment) ambiguity set (Jin et al., 26 Aug 2025). HO can be rewritten as minimization over distributions $$\min_{x \in \mathcal{X}}~ \frac{1}{n}\sum_{i=1}^n f(x, \xi_i).$$9 for $R(x)$0: $R(x)$1 with $R(x)$2.

Key theoretical developments:

• Optimal choice $R(x)$3 yields asymptotic optimality and $R(x)$4 bias.
• Finite-sample guarantees: Explicit nonasymptotic probabilistic bounds on the ambiguity coverage under mild tail conditions.
• Tractable reformulation: For problems with piecewise-linear convexity in $R(x)$5, HO reduces to a single LP or SDP.

HO also provides a scenario reduction mechanism: when only $R(x)$6 scenarios are retained, a recalibration $R(x)$7 ensures solution robustness against information loss.

5. Algorithmic and Computational Aspects

Regularized SAA methods, depending on the regularizer structure, admit tractable algorithmic reformulations:

Method	Key Reformulation	Typical Solver/Complexity
Low-rank RSAA	S³ONC constrained opt.	Trust-region/Cubic reg. (local)
Scenario-based RSAA	MISOCP/SOCP	CPLEX, MOSEK, $R(x)$8 variables
HO (moment-DRO)	LP/SDP	Standard LP/SDP solvers

For large $R(x)$9 in low-rank RSAA, warm-starting at the nuclear-norm solution and local refinement with second-order methods suffices. SBR-SAA leverages scenario clustering and exact weights via linear programming to ensure compact regularization that is still statistically effective. In HO, scenario reduction directly accelerates the bottleneck by reducing the SAA size and robustifying against discarded information.

6. Performance Guarantees and Practical Implications

RSAA methods provide provable finite-sample and asymptotic guarantees under mild conditions (e.g., light-tails, mild smoothness, low-rank structure). In high-dimensional settings with intrinsic structure, RSAA can reduce sample complexity by orders of magnitude over classical SAA (Liu et al., 2019). SBR-SAA balances computational feasibility and robustness, matching or exceeding the out-of-sample performance of pure SAA and WDRO in diverse applications including multi-product newsvendor and portfolio optimization (Fonseca et al., 20 Nov 2025). HO demonstrates superior empirical and theoretical performance versus both extremes (SAA and DRO), with robust behavior across diverse sample sizes, and adapts seamlessly to scenario reduction (Jin et al., 26 Aug 2025).

A plausible implication is that RSAA and its variants are foundational for scalable stochastic programming under modern data regimes, particularly when uncertainty structure, robustness, and efficiency are all first-class requirements.

7. Extensions and Open Directions

Open research avenues include:

• Extension of RSAA to non-PSD constraint domains and nonconvex loss landscapes.
• Sharper dependence on true rank $$\min_{x\in\mathcal{X}}~ \frac{1}{n}\sum_{i=1}^n f(x, \xi_i) + \lambda R(x),$$0 in high-dimensional RSAA.
• Exploration of alternative regularizers (e.g., Schatten-$$\min_{x\in\mathcal{X}}~ \frac{1}{n}\sum_{i=1}^n f(x, \xi_i) + \lambda R(x),$$1 norms, group-sparsity, tail risk metrics).
• Deeper understanding of scenario selection strategies balancing computational cost and statistical robustness, especially under adversarial data regimes.
• Improved scenario reduction algorithms harmonized with HO.

These directions reflect the centrality of regularization in bridging SAA's empirical fit and DRO's safety/robustness, enabling effective, theory-backed solutions to high-dimensional and risk-sensitive stochastic optimization.