MARS-M: Optimizer & Recommender Variants

• MARS-M is a context-dependent term used in optimization, sequential recommendation, and memory-agent configurations, each with distinct definitions and applications.
• In optimization, MARS-M integrates variance reduction with matrix preconditioning, enabling faster convergence and improved training of large-scale neural networks.
• Practical implementations balance exact and approximate methods to sustain low computational cost while demonstrating empirical gains in language modeling and computer vision.

Searching arXiv for exact phrase and nearby uses of “MARS-M” to disambiguate the term and verify the most relevant papers. MARS-M is an arXiv label with multiple technical uses. The exact string appears explicitly in "MARS-M: When Variance Reduction Meets Matrices" (Liu et al., 20 Oct 2025), where it denotes a matrix-based optimizer that integrates the variance reduction technique in MARS with Muon, and in a sequential recommendation paper where it denotes the Mamba-based instantiation of a recency-aggregation framework (Yu et al., 2 Jun 2026). In several other MARS papers, by contrast, the string does not appear explicitly and functions only as an inferred shorthand for a memory-enabled or multi-agent variant (Liang et al., 25 Mar 2025). In current optimization literature, MARS-M refers to a method designed to combine matrix-level preconditioning with variance-reduced stochastic updates for training large-scale neural networks, including LLMs (Liu et al., 20 Oct 2025).

1. Terminology, naming, and scope

The term MARS-M is not globally uniform across the arXiv corpus. In optimization, it is the formal name of a method whose title explicitly states its conceptual synthesis: variance reduction meeting matrix-based optimization (Liu et al., 20 Oct 2025). In sequential recommendation, MARS-M is the Mamba-based member of a dual-instantiation framework, paired with MARS-T for Transformers (Yu et al., 2 Jun 2026). In contrast, the 2025 agent paper "MARS: Memory-Enhanced Agents with Reflective Self-improvement" does not use the exact string; there, “MARS-M” is a natural interpretation for the configuration with MemorySyntax enabled, corresponding to the “w memo” ablation rather than an official variant name (Liang et al., 25 Mar 2025).

This ambiguity matters because the underlying objects differ fundamentally. The optimizer MARS-M operates on matrix parameters and stochastic gradients during model training. The recommender MARS-M is a Mamba-based sequential encoder with explicit multi-rate recency aggregation. The memory-agent usage is an editorial shorthand for a memory-enabled configuration, not a named algorithmic object in the source paper. For technical precision, discussion of MARS-M therefore depends entirely on domain context.

2. Optimizer formulation: variance reduction integrated with matrix preconditioning

In "MARS-M: When Variance Reduction Meets Matrices" (Liu et al., 20 Oct 2025), the starting point is the contrast between scalar-based adaptive methods such as AdamW and matrix-based preconditioned optimizers such as Muon. Scalar methods maintain elementwise first and second moments and therefore act with a diagonal preconditioner, whereas Muon treats each weight matrix as a matrix and uses gradient orthogonalization. In its basic form, Muon updates a momentum matrix

$$\mathbf{M}_t = \beta \mathbf{M}_{t-1} + \nabla f(\mathbf{X}_t,\xi_t),$$

computes an approximate orthogonal factor

$$\mathbf{O}_t = \mathrm{NewtonSchulz}(\mathbf{M}_t) \approx \mathbf{U}_t \mathbf{V}_t^\top,$$

and then steps in that orthogonalized direction. The practical Muon variant used in large-scale language-model training is Moonlight, which adds scaling and decoupled weight decay (Liu et al., 20 Oct 2025).

MARS contributes a different ingredient: a scaled control-variate correction derived from variance-reduction methods such as STORM. MARS-M combines these two ideas by applying the MARS correction to the matrix gradient before Muon-style preconditioning. For a matrix parameter $\mathbf{X}_t \in \mathbb{R}^{m \times n}$ and objective

$F(\mathbf{X}) = \mathbb{E}_\xi[f(\mathbf{X},\xi)],$

the exact corrected gradient is

$$\mathbf{C}_t = \nabla f(\mathbf{X}_t,\xi_t) + \gamma_t \left(\frac{\beta}{1-\beta}\right) \big( \nabla f(\mathbf{X}_t,\xi_t) - \nabla f(\mathbf{X}_{t-1},\xi_t) \big).$$

This is then clipped in Frobenius norm,

$$\mathrm{Clip}(\mathbf{C}_t,1)= \begin{cases} \mathbf{C}_t/\|\mathbf{C}_t\|_F, & \|\mathbf{C}_t\|_F > 1,\ \mathbf{C}_t, & \text{otherwise}, \end{cases}$$

and used to update the momentum matrix

$$\mathbf{M}_t = \beta \mathbf{M}_{t-1} + (1-\beta)\,\mathrm{Clip}(\mathbf{C}_t,1).$$

The preconditioned direction is then obtained by Newton–Schulz iteration,

$$\mathbf{O}_t = \mathrm{NewtonSchulz}(\mathbf{M}_t) \approx \mathbf{U}_t \mathbf{V}_t^\top,$$

and the Moonlight-style step becomes

$$\mathbf{X}_{t+1} = \mathbf{X}_t - \eta_t\Big( 0.2 \cdot \mathbf{O}_t \cdot \sqrt{\max(m,n)} + \lambda \mathbf{X}_t \Big).$$

The algorithm is therefore a direct composition: MARS on the gradient pathway, Muon/Moonlight on the preconditioning pathway (Liu et al., 20 Oct 2025).

3. Approximate MARS-M and practical implementation

The exact formulation requires two gradient evaluations on the same minibatch, $\nabla f(\mathbf{X}_t,\xi_t)$ and $$\mathbf{O}_t = \mathrm{NewtonSchulz}(\mathbf{M}_t) \approx \mathbf{U}_t \mathbf{V}_t^\top,$$0, which would roughly double backward-pass cost. The practical variant used for LLMs replaces the second quantity by the previous gradient on the previous batch, yielding

$$\mathbf{O}_t = \mathrm{NewtonSchulz}(\mathbf{M}_t) \approx \mathbf{U}_t \mathbf{V}_t^\top,$$1

This preserves the single-gradient training cost of Moonlight while retaining the MARS-style correction (Liu et al., 20 Oct 2025).

Algebraically, the paper shows that the approximate formulation can be rewritten as a Moonlight-style recursion with modified momentum coefficients: $$\mathbf{O}_t = \mathrm{NewtonSchulz}(\mathbf{M}_t) \approx \mathbf{U}_t \mathbf{V}_t^\top,$$2

$$\mathbf{O}_t = \mathrm{NewtonSchulz}(\mathbf{M}_t) \approx \mathbf{U}_t \mathbf{V}_t^\top,$$3

followed by the same Newton–Schulz and update equations used in Moonlight. This recasting is practically important because it explains why approximate MARS-M can be implemented with low engineering overhead inside existing Muon/Moonlight codepaths (Liu et al., 20 Oct 2025).

The optimizer is applied only to matrix-like parameters. Vector-like parameters and embeddings are still trained with AdamW at the same learning rate, mirroring standard Moonlight practice. In experiments, the paper uses $$\mathbf{O}_t = \mathrm{NewtonSchulz}(\mathbf{M}_t) \approx \mathbf{U}_t \mathbf{V}_t^\top,$$4, a clipping threshold of $$\mathbf{O}_t = \mathrm{NewtonSchulz}(\mathbf{M}_t) \approx \mathbf{U}_t \mathbf{V}_t^\top,$$5, and small constant values of $$\mathbf{O}_t = \mathrm{NewtonSchulz}(\mathbf{M}_t) \approx \mathbf{U}_t \mathbf{V}_t^\top,$$6, typically $$\mathbf{O}_t = \mathrm{NewtonSchulz}(\mathbf{M}_t) \approx \mathbf{U}_t \mathbf{V}_t^\top,$$7 or $$\mathbf{O}_t = \mathrm{NewtonSchulz}(\mathbf{M}_t) \approx \mathbf{U}_t \mathbf{V}_t^\top,$$8, while leaving Newton–Schulz, scaling by $$\mathbf{O}_t = \mathrm{NewtonSchulz}(\mathbf{M}_t) \approx \mathbf{U}_t \mathbf{V}_t^\top,$$9, and weight decay unchanged (Liu et al., 20 Oct 2025).

4. Convergence theory and rate improvement

The theoretical analysis is carried out for nonconvex stochastic optimization under standard assumptions: unbiased stochastic gradients, bounded variance,

$\mathbf{X}_t \in \mathbb{R}^{m \times n}$0

and $\mathbf{X}_t \in \mathbb{R}^{m \times n}$1-smoothness,

$\mathbf{X}_t \in \mathbb{R}^{m \times n}$2

Under these assumptions, the paper proves a nonconvex convergence guarantee for exact MARS-M with $\mathbf{X}_t \in \mathbb{R}^{m \times n}$3, learning rate

$\mathbf{X}_t \in \mathbb{R}^{m \times n}$4

and momentum schedule

$\mathbf{X}_t \in \mathbb{R}^{m \times n}$5

together with a theoretically chosen $\mathbf{X}_t \in \mathbb{R}^{m \times n}$6 (Liu et al., 20 Oct 2025).

The resulting average-gradient bound is of order

$\mathbf{X}_t \in \mathbb{R}^{m \times n}$7

improving on the $\mathbf{X}_t \in \mathbb{R}^{m \times n}$8 rate cited for Muon. The proof introduces a tracking error

$\mathbf{X}_t \in \mathbb{R}^{m \times n}$9

and shows that the MARS correction contributes a negative variance-reduction term in the recursion for $F(\mathbf{X}) = \mathbb{E}_\xi[f(\mathbf{X},\xi)],$0. This term tightens the Lyapunov descent argument and yields the sharper $F(\mathbf{X}) = \mathbb{E}_\xi[f(\mathbf{X},\xi)],$1 dependence (Liu et al., 20 Oct 2025).

Conceptually, the significance of the theorem is not only that MARS-M improves the asymptotic rate, but also that it does so without abandoning the matrix-orthogonalization structure that makes Muon attractive for large matrices. The method therefore advances both the optimizer’s empirical design and its theoretical status.

5. Empirical results on language modeling and vision

The language-model experiments use GPT-2 style transformers at four scales—125M, 355M, 770M, and 1.5B—trained for 100k steps with context length 1024 on OpenWebText and FineWeb-Edu 100B. The setup uses 16 NVIDIA H800 GPUs for small models and 32 NVIDIA H800 GPUs for larger ones. Across these runs, MARS-M yields consistently lower training and validation losses than Moonlight, with the gap particularly visible in later training stages (Liu et al., 20 Oct 2025).

Downstream evaluation is performed on ARC, HellaSwag, MMLU, OpenBookQA, PIQA, SciQ, and WinoGrande, using 0-shot and 2-shot accuracy. For GPT-2 XL on FineWeb-Edu 100B in the 2-shot setting, Moonlight attains an average score of $F(\mathbf{X}) = \mathbb{E}_\xi[f(\mathbf{X},\xi)],$2, while MARS-M reaches $F(\mathbf{X}) = \mathbb{E}_\xi[f(\mathbf{X},\xi)],$3 with $F(\mathbf{X}) = \mathbb{E}_\xi[f(\mathbf{X},\xi)],$4 and $F(\mathbf{X}) = \mathbb{E}_\xi[f(\mathbf{X},\xi)],$5 with $F(\mathbf{X}) = \mathbb{E}_\xi[f(\mathbf{X},\xi)],$6. Individual benchmark improvements include ARC-C $F(\mathbf{X}) = \mathbb{E}_\xi[f(\mathbf{X},\xi)],$7, HellaSwag $F(\mathbf{X}) = \mathbb{E}_\xi[f(\mathbf{X},\xi)],$8, SciQ $F(\mathbf{X}) = \mathbb{E}_\xi[f(\mathbf{X},\xi)],$9, and WinoGrande $$\mathbf{C}_t = \nabla f(\mathbf{X}_t,\xi_t) + \gamma_t \left(\frac{\beta}{1-\beta}\right) \big( \nabla f(\mathbf{X}_t,\xi_t) - \nabla f(\mathbf{X}_{t-1},\xi_t) \big).$$0 for the $$\mathbf{C}_t = \nabla f(\mathbf{X}_t,\xi_t) + \gamma_t \left(\frac{\beta}{1-\beta}\right) \big( \nabla f(\mathbf{X}_t,\xi_t) - \nabla f(\mathbf{X}_{t-1},\xi_t) \big).$$1 setting (Liu et al., 20 Oct 2025).

The computer-vision experiments use ResNet-18 on CIFAR-10 for 200 epochs with batch size 128. Here the paper compares Moonlight, exact MARS-M, and approximate MARS-M. Exact MARS-M achieves the lowest test loss and the highest test accuracy; approximate MARS-M remains better than Moonlight but slightly worse than the exact variant. This split is important because it isolates the effect of true same-batch variance reduction from the cheaper approximation used in language-model training (Liu et al., 20 Oct 2025).

An ablation over $$\mathbf{C}_t = \nabla f(\mathbf{X}_t,\xi_t) + \gamma_t \left(\frac{\beta}{1-\beta}\right) \big( \nabla f(\mathbf{X}_t,\xi_t) - \nabla f(\mathbf{X}_{t-1},\xi_t) \big).$$2 on GPT-2 small shows that overly large $$\mathbf{C}_t = \nabla f(\mathbf{X}_t,\xi_t) + \gamma_t \left(\frac{\beta}{1-\beta}\right) \big( \nabla f(\mathbf{X}_t,\xi_t) - \nabla f(\mathbf{X}_{t-1},\xi_t) \big).$$3 hurts performance, while values in the range $$\mathbf{C}_t = \nabla f(\mathbf{X}_t,\xi_t) + \gamma_t \left(\frac{\beta}{1-\beta}\right) \big( \nabla f(\mathbf{X}_t,\xi_t) - \nabla f(\mathbf{X}_{t-1},\xi_t) \big).$$4 to $$\mathbf{C}_t = \nabla f(\mathbf{X}_t,\xi_t) + \gamma_t \left(\frac{\beta}{1-\beta}\right) \big( \nabla f(\mathbf{X}_t,\xi_t) - \nabla f(\mathbf{X}_{t-1},\xi_t) \big).$$5 behave similarly and robustly. This is consistent with the broader MARS observation that full-strength control-variate correction is too aggressive for deep-learning practice, whereas small, scaled corrections improve stability and speed (Liu et al., 20 Oct 2025).

6. Other documented meanings of “MARS-M”

Outside optimization, the exact string MARS-M also appears in sequential recommendation. In "MARS: Multi-rate Aggregation of Recency Signals for Sequential Recommendation across Sparse and Dense Regimes" (Yu et al., 2 Jun 2026), MARS-M is the Mamba-based instantiation of a timestamp-aware recency aggregation module. It is selected automatically for dense datasets according to the rule

$$\mathbf{C}_t = \nabla f(\mathbf{X}_t,\xi_t) + \gamma_t \left(\frac{\beta}{1-\beta}\right) \big( \nabla f(\mathbf{X}_t,\xi_t) - \nabla f(\mathbf{X}_{t-1},\xi_t) \big).$$6

and on ML-1M it reports HR@10 $$\mathbf{C}_t = \nabla f(\mathbf{X}_t,\xi_t) + \gamma_t \left(\frac{\beta}{1-\beta}\right) \big( \nabla f(\mathbf{X}_t,\xi_t) - \nabla f(\mathbf{X}_{t-1},\xi_t) \big).$$7 versus $$\mathbf{C}_t = \nabla f(\mathbf{X}_t,\xi_t) + \gamma_t \left(\frac{\beta}{1-\beta}\right) \big( \nabla f(\mathbf{X}_t,\xi_t) - \nabla f(\mathbf{X}_{t-1},\xi_t) \big).$$8 for SIGMA, together with $$\mathbf{C}_t = \nabla f(\mathbf{X}_t,\xi_t) + \gamma_t \left(\frac{\beta}{1-\beta}\right) \big( \nabla f(\mathbf{X}_t,\xi_t) - \nabla f(\mathbf{X}_{t-1},\xi_t) \big).$$9 fewer MFLOPs (Yu et al., 2 Jun 2026). In that literature, MARS-M therefore has nothing to do with variance reduction or Muon; it is a Mamba-based recommender.

A different situation holds in the multi-agent memory paper "MARS: Memory-Enhanced Agents with Reflective Self-improvement" (Liang et al., 25 Mar 2025). There, the exact string does not appear explicitly. The closest interpretation is “MARS with its MemorySyntax module enabled,” corresponding to the “w memo” rows in the ablation table rather than an official named variant. The same pattern recurs in "MARS: Optimizing Dual-System Deep ReSearch via Multi-Agent Reinforcement Learning" (Chen et al., 6 Oct 2025), where the paper states that it never explicitly writes “MARS-M” and uses the label only as an informed extrapolation for the multi-agent reinforcement-learning component (Chen et al., 6 Oct 2025).

A common misconception is therefore to treat MARS-M as a single, portable acronym. The arXiv record does not support that reading. It supports, instead, a family of context-dependent usages: an explicitly named optimizer (Liu et al., 20 Oct 2025), an explicitly named Mamba recommender (Yu et al., 2 Jun 2026), and several inferred shorthands in other MARS frameworks (Liang et al., 25 Mar 2025). Among these, the optimizer is the only case where “MARS-M” is itself the formal title of the method.