Multi-Gradient Stochastic Mirror Descent

• MSMD is an optimization framework that extends classical stochastic mirror descent by integrating matrix-parameter and multi-objective gradients using Bregman divergences.
• It employs dual-space and proximal updates with tailored mirror maps to enforce implicit regularization and achieve fast convergence in high-dimensional settings.
• MSMD demonstrates robust empirical performance in applications such as multi-class prediction and multi-task learning, offering scalable solutions for complex stochastic optimization.

Multi-Gradient Stochastic Mirror Descent (MSMD) refers to a class of optimization algorithms that combine mirror descent techniques with multi-gradient or matrix-parameter structures, and extend naturally to stochastic and multi-objective settings. Two main research threads have crystallized in the literature: the matrix-parameter SMD framework for multi-output learning (Akhtiamov et al., 22 Feb 2026), and the multi-objective saddle-point MSMD approach for Pareto optimization and multi-task training (Yang et al., 2024). Both paradigms build on the use of Bregman divergences and mirror mappings to induce specific implicit bias and handle high-dimensional, multi-channel or multi-objective systems.

1. Formal Definitions and Mathematical Framework

The MSMD framework generalizes classical stochastic mirror descent by managing either matrices as model parameters (typical in classification or matrix completion) or multi-objective losses using simultaneous gradients.

Matrix-Parameter MSMD

Given $W \in \mathbb{R}^{d \times k}$, the goal is to interpolate a set of linear measurements $A(W) = b \in \mathbb{R}^p$ with loss functions $\ell_i: \mathbb{R}\to\mathbb{R}_{\ge0}$ for $i=1,\dots,p$, where $A$ is a linear operator with rows written as vectorizations $a_i \in \mathbb{R}^{d\times k}$. The empirical risk is:

$$L(W) = \frac{1}{p} \sum_{i=1}^p \ell_i(A(W)_i - b_i)$$

In the overparameterized regime ($dk>p$), the solution set is infinite; MSMD selects a distinguished element via its implicit bias (Akhtiamov et al., 22 Feb 2026).

Multi-Objective MSMD

For stochastic multi-objective optimization (SMOO) with $m$ objectives $F(x) = (f_1(x),\ldots,f_m(x))^T$, a Pareto-stationary point is sought. The algorithm solves the saddle-point problem:

$A(W) = b \in \mathbb{R}^p$0

by simultaneous mirror steps on $A(W) = b \in \mathbb{R}^p$1 and $A(W) = b \in \mathbb{R}^p$2 using stochastic gradient information (Yang et al., 2024).

Both threads leverage Bregman divergences, determined by a strongly convex mirror map $A(W) = b \in \mathbb{R}^p$3. The choice of $A(W) = b \in \mathbb{R}^p$4 induces different implicit regularization properties.

2. Algorithmic Structure and Update Rules

Matrix-Parameter MSMD Updates

For matrix parameters, MSMD proceeds as follows:

• Dual-space update:

$A(W) = b \in \mathbb{R}^p$5

where $A(W) = b \in \mathbb{R}^p$6 is the stochastic gradient in $A(W) = b \in \mathbb{R}^p$7-space.

• Primal (proximal) update:

$A(W) = b \in \mathbb{R}^p$8

These representations are equivalent under Legendre $A(W) = b \in \mathbb{R}^p$9.

• Projection interpretation:

$\ell_i: \mathbb{R}\to\mathbb{R}_{\ge0}$0

Multi-Objective MSMD Structure

The MSMD method for SMOO (Yang et al., 2024) consists of a double-loop algorithm:

• Inner loop: Uses SMD to solve the primal-dual saddle-point subproblem for $\ell_i: \mathbb{R}\to\mathbb{R}_{\ge0}$1, where $\ell_i: \mathbb{R}\to\mathbb{R}_{\ge0}$2 is a descent direction and $\ell_i: \mathbb{R}\to\mathbb{R}_{\ge0}$3 is a vector of objective weights over the simplex $\ell_i: \mathbb{R}\to\mathbb{R}_{\ge0}$4. Mirror maps for $\ell_i: \mathbb{R}\to\mathbb{R}_{\ge0}$5 and $\ell_i: \mathbb{R}\to\mathbb{R}_{\ge0}$6 may differ (typically Euclidean for $\ell_i: \mathbb{R}\to\mathbb{R}_{\ge0}$7, entropy for $\ell_i: \mathbb{R}\to\mathbb{R}_{\ge0}$8).
• Outer loop: Updates $\ell_i: \mathbb{R}\to\mathbb{R}_{\ge0}$9 as $i=1,\dots,p$0, where $i=1,\dots,p$1 is a weighted average of recent inner-loop iterates.
• Pseudocode summary:

| Variable | Description | Mirror Map | |-------------------|-------------------------------------------------|------------------------------| | $i=1,\dots,p$2 | Descent direction | $i=1,\dots,p$3| | $i=1,\dots,p$4 | Objective weighting vector ($i=1,\dots,p$5) | $i=1,\dots,p$6 | | $i=1,\dots,p$7 | Model parameter | Problem dependent |

This allows per-iteration sampling (one gradient sample per inner step), controlling per-iteration cost and variance.

3. Convergence Properties and Theoretical Guarantees

Matrix MSMD: Exponential Convergence and Implicit Bias

Under standing assumptions (strong convexity of $i=1,\dots,p$8 and $i=1,\dots,p$9, unbiased sampling, $A$0 full row-rank, suitable step-size), MSMD yields:

• Almost sure convergence: $A$1 and $A$2.
• Exponential in-expectation rate:

$A$3

for $A$4 on a compact Bregman ball $A$5.

Implicit bias: In the interpolating regime, MSMD selects the unique interpolator minimizing $A$6.

Multi-Objective MSMD: Sublinear Convergence

For SMOO, MSMD achieves sublinear convergence in expected directional norm:

• Convergence rates:
  • For fixed $A$7, fixed inner stepsizes:

  $A$8 - With variable stepsizes:

  $A$9

Error decomposition relies on controlling the bias from the saddle-point mapping of $a_i \in \mathbb{R}^{d\times k}$0 and $a_i \in \mathbb{R}^{d\times k}$1 via Bregman-divergence geometry. The method admits rigorous convergence proofs using primal-dual gap bounds and descent lemmas (Yang et al., 2024).

4. Choice of Mirror Maps and Induced Regularization

The mirror map $a_i \in \mathbb{R}^{d\times k}$2 encodes geometry and regularization:

• Euclidean mirror: $a_i \in \mathbb{R}^{d\times k}$3. $a_i \in \mathbb{R}^{d\times k}$4, bias yields minimum Frobenius norm interpolator.
• Schatten-$a_i \in \mathbb{R}^{d\times k}$5 mirror: $a_i \in \mathbb{R}^{d\times k}$6$a_i \in \mathbb{R}^{d\times k}$7. Encourages low-rank $a_i \in \mathbb{R}^{d\times k}$8 in the matrix completion setup.
• Log-det mirror: $a_i \in \mathbb{R}^{d\times k}$9 (for square $$L(W) = \frac{1}{p} \sum_{i=1}^p \ell_i(A(W)_i - b_i)$$0). Bias towards maximal-volume solutions (used in D-optimal design).

The multi-objective SMOO variant uses standard mirrors: squared 2-norm for $$L(W) = \frac{1}{p} \sum_{i=1}^p \ell_i(A(W)_i - b_i)$$1 and negative entropy for $$L(W) = \frac{1}{p} \sum_{i=1}^p \ell_i(A(W)_i - b_i)$$2 (simplex), enabling analytical projection and efficient SMD updates.

5. Illustrative Examples and Benchmarks

Matrix-Parameter Toy Example

For $$L(W) = \frac{1}{p} \sum_{i=1}^p \ell_i(A(W)_i - b_i)$$3, $$L(W) = \frac{1}{p} \sum_{i=1}^p \ell_i(A(W)_i - b_i)$$4, and a single data point $$L(W) = \frac{1}{p} \sum_{i=1}^p \ell_i(A(W)_i - b_i)$$5, with squared loss:

• Euclidean MSMD: Solution is minimum-norm $$L(W) = \frac{1}{p} \sum_{i=1}^p \ell_i(A(W)_i - b_i)$$6 such that $$L(W) = \frac{1}{p} \sum_{i=1}^p \ell_i(A(W)_i - b_i)$$7.
• Schatten-1.05 MSMD: Solution $$L(W) = \frac{1}{p} \sum_{i=1}^p \ell_i(A(W)_i - b_i)$$8 still interpolates but has minimal approximate nuclear norm among such. Practically, applying Schatten-$$L(W) = \frac{1}{p} \sum_{i=1}^p \ell_i(A(W)_i - b_i)$$9 mirrors to rank-deficient feature matrices can yield rank-deficient (strongly regularized) solutions, in contrast to Euclidean mirrors.

SMOO/Multi-Task Learning

Extensive experiments spanning classic test functions (e.g., BK1, FF1, Lov1, MOP5) and multi-task learning with multi-MNIST:

• Pareto fronts: MSMD produces more complete/stable fronts under noise, capturing extremes missed by alternatives such as CR-MOGM or SDMGrad.
• Multi-task networks (e.g., CNN on Multi-MNIST): MSMD achieves the lowest training loss and highest Top-5 accuracy, with competitive Top-1 accuracy and lower computational cost (one sample per inner iteration vs. three for SDMGrad) (Yang et al., 2024).

A plausible implication is that the Bregman geometry and reduced per-step sampling cost make MSMD favorable for high-noise, high-dimensional, or multi-task regimes.

6. Extensions, Variants, and Applications

Preference-based MSMD

When explicit user-specified objectives are desired (e.g., weighted-sum preferences), MSMD admits an efficient extension. By incorporating a preference vector $dk>p$0 and regularizing the subproblem with $dk>p$1, the method adapts to user priorities while retaining convergence guarantees, up to changes in step sizes and variance constants.

Application Domains

• Multi-class and multi-output prediction (Akhtiamov et al., 22 Feb 2026)
• Matrix completion and low-rank recovery
• Multi-objective optimization for learning and control (Yang et al., 2024)
• Multi-task neural network training

The flexibility in mirror map choice enables MSMD methods to interpolate between classical regularization regimes, soft-thresholding, and geometric coverage.

7. Summary of Theoretical and Practical Significance

MSMD generalizes stochastic mirror descent to matrix/multi-gradient and multi-objective settings, providing:

• Strong theoretical guarantees (exponential and sublinear rates under suitable assumptions)
• Bregman divergence-driven implicit bias for structured solutions
• Scalability and efficiency in high-dimensional stochastic settings due to per-iteration sampling economy
• Superior or competitive empirical performance on standard benchmarks, including robust Pareto superior fronts and multi-task accuracy in neural architectures

These properties unify numerous regularization and geometric phenomena within a single algorithmic and analytical framework, with direct implications for overparameterized, high-dimensional learning systems (Akhtiamov et al., 22 Feb 2026, Yang et al., 2024).