---
title: 'MARS-M: Optimizer & Recommender Variants'
url: https://www.emergentmind.com/topics/mars-m
type: topic
---

# MARS-M: Optimizer & Recommender Variants

Searching arXiv for exact phrase and nearby uses of “MARS-M” to disambiguate the term and verify the most relevant papers.
MARS-M is an arXiv label with multiple technical uses. The exact string appears explicitly in "MARS-M: When Variance Reduction Meets Matrices" [2510.21800], where it denotes a matrix-based optimizer that integrates the variance reduction technique in MARS with Muon, and in a sequential recommendation paper where it denotes the Mamba-based instantiation of a recency-aggregation framework [2606.03718]. In several other MARS papers, by contrast, the string does not appear explicitly and functions only as an inferred shorthand for a memory-enabled or multi-agent variant [2503.19271]. In current optimization literature, MARS-M refers to a method designed to combine matrix-level preconditioning with variance-reduced stochastic updates for training large-scale neural networks, including large language models [2510.21800].

## 1. Terminology, naming, and scope

The term MARS-M is not globally uniform across the arXiv corpus. In optimization, it is the formal name of a method whose title explicitly states its conceptual synthesis: variance reduction meeting matrix-based optimization [2510.21800]. In sequential recommendation, MARS-M is the Mamba-based member of a dual-instantiation framework, paired with MARS-T for Transformers [2606.03718]. In contrast, the 2025 agent paper "MARS: Memory-Enhanced Agents with Reflective Self-improvement" does not use the exact string; there, “MARS-M” is a natural interpretation for the configuration with MemorySyntax enabled, corresponding to the “w memo” ablation rather than an official variant name [2503.19271].

This ambiguity matters because the underlying objects differ fundamentally. The optimizer MARS-M operates on matrix parameters and stochastic gradients during model training. The recommender MARS-M is a Mamba-based sequential encoder with explicit multi-rate recency aggregation. The memory-agent usage is an editorial shorthand for a memory-enabled configuration, not a named algorithmic object in the source paper. For technical precision, discussion of MARS-M therefore depends entirely on domain context.

## 2. Optimizer formulation: variance reduction integrated with matrix preconditioning

In "MARS-M: When Variance Reduction Meets Matrices" [2510.21800], the starting point is the contrast between scalar-based adaptive methods such as AdamW and matrix-based preconditioned optimizers such as Muon. Scalar methods maintain elementwise first and second moments and therefore act with a diagonal preconditioner, whereas Muon treats each weight matrix as a matrix and uses gradient orthogonalization. In its basic form, Muon updates a momentum matrix
\[
\mathbf{M}_t = \beta \mathbf{M}_{t-1} + \nabla f(\mathbf{X}_t,\xi_t),
\]
computes an approximate orthogonal factor
\[
\mathbf{O}_t = \mathrm{NewtonSchulz}(\mathbf{M}_t) \approx \mathbf{U}_t \mathbf{V}_t^\top,
\]
and then steps in that orthogonalized direction. The practical Muon variant used in large-scale language-model training is Moonlight, which adds scaling and decoupled weight decay [2510.21800].

MARS contributes a different ingredient: a scaled control-variate correction derived from variance-reduction methods such as STORM. MARS-M combines these two ideas by applying the MARS correction to the matrix gradient before Muon-style preconditioning. For a matrix parameter \(\mathbf{X}_t \in \mathbb{R}^{m \times n}\) and objective
\[
F(\mathbf{X}) = \mathbb{E}_\xi[f(\mathbf{X},\xi)],
\]
the exact corrected gradient is
\[
\mathbf{C}_t = \nabla f(\mathbf{X}_t,\xi_t)
+ \gamma_t \left(\frac{\beta}{1-\beta}\right)
\big(
\nabla f(\mathbf{X}_t,\xi_t) - \nabla f(\mathbf{X}_{t-1},\xi_t)
\big).
\]
This is then clipped in Frobenius norm,
\[
\mathrm{Clip}(\mathbf{C}_t,1)=
\begin{cases}
\mathbf{C}_t/\|\mathbf{C}_t\|_F, & \|\mathbf{C}_t\|_F > 1,\\
\mathbf{C}_t, & \text{otherwise},
\end{cases}
\]
and used to update the momentum matrix
\[
\mathbf{M}_t = \beta \mathbf{M}_{t-1} + (1-\beta)\,\mathrm{Clip}(\mathbf{C}_t,1).
\]
The preconditioned direction is then obtained by Newton–Schulz iteration,
\[
\mathbf{O}_t = \mathrm{NewtonSchulz}(\mathbf{M}_t) \approx \mathbf{U}_t \mathbf{V}_t^\top,
\]
and the Moonlight-style step becomes
\[
\mathbf{X}_{t+1}
=
\mathbf{X}_t
-
\eta_t\Big(
0.2 \cdot \mathbf{O}_t \cdot \sqrt{\max(m,n)}
+
\lambda \mathbf{X}_t
\Big).
\]
The algorithm is therefore a direct composition: MARS on the gradient pathway, Muon/Moonlight on the preconditioning pathway [2510.21800].

## 3. Approximate MARS-M and practical implementation

The exact formulation requires two gradient evaluations on the same minibatch, \(\nabla f(\mathbf{X}_t,\xi_t)\) and \(\nabla f(\mathbf{X}_{t-1},\xi_t)\), which would roughly double backward-pass cost. The practical variant used for large language models replaces the second quantity by the previous gradient on the previous batch, yielding
\[
\mathbf{C}_t = \nabla f(\mathbf{X}_t,\xi_t)
+
\gamma_t\left(\frac{\beta}{1-\beta}\right)
\big(
\nabla f(\mathbf{X}_t,\xi_t) - \nabla f(\mathbf{X}_{t-1},\xi_{t-1})
\big).
\]
This preserves the single-gradient training cost of Moonlight while retaining the MARS-style correction [2510.21800].

Algebraically, the paper shows that the approximate formulation can be rewritten as a Moonlight-style recursion with modified momentum coefficients:
\[
\mathbf{U}_t
=
\beta \mathbf{U}_{t-1}
+
\frac{(1-\gamma_t)(1-\beta)}{\beta}
\nabla f(\mathbf{X}_t,\xi_t),
\]
\[
\mathbf{M}_t
=
\beta \mathbf{U}_t + \gamma_t \nabla f(\mathbf{X}_t,\xi_t),
\]
followed by the same Newton–Schulz and update equations used in Moonlight. This recasting is practically important because it explains why approximate MARS-M can be implemented with low engineering overhead inside existing Muon/Moonlight codepaths [2510.21800].

The optimizer is applied only to matrix-like parameters. Vector-like parameters and embeddings are still trained with AdamW at the same learning rate, mirroring standard Moonlight practice. In experiments, the paper uses \(\beta = 0.95\), a clipping threshold of \(1\), and small constant values of \(\gamma\), typically \(0.01\) or \(0.025\), while leaving Newton–Schulz, scaling by \(0.2\sqrt{\max(m,n)}\), and weight decay unchanged [2510.21800].

## 4. Convergence theory and rate improvement

The theoretical analysis is carried out for nonconvex stochastic optimization under standard assumptions: unbiased stochastic gradients, bounded variance,
\[
\mathbb{E}\big[\|\nabla f(\mathbf{X},\xi)-\nabla F(\mathbf{X})\|_F^2\big]\le \sigma^2,
\]
and \(L\)-smoothness,
\[
\|\nabla f(\mathbf{X},\xi)-\nabla f(\mathbf{Y},\xi)\|_F
\le
L\|\mathbf{X}-\mathbf{Y}\|_F.
\]
Under these assumptions, the paper proves a nonconvex convergence guarantee for exact MARS-M with \(\lambda=0\), learning rate
\[
\eta_t = (s+t)^{-2/3},
\]
and momentum schedule
\[
\beta_{t+1}=1-2\eta_t,
\]
together with a theoretically chosen \(\gamma_{t+1}\) [2510.21800].

The resulting average-gradient bound is of order
\[
\frac{1}{T}\sum_{t=1}^T \mathbb{E}\|\nabla F(\mathbf{X}_t)\|_F
=
\tilde{\mathcal{O}}(T^{-1/3}),
\]
improving on the \(\tilde{\mathcal{O}}(T^{-1/4})\) rate cited for Muon. The proof introduces a tracking error
\[
e_t := \nabla F(\mathbf{X}_t) - \mathbf{M}_t
\]
and shows that the MARS correction contributes a negative variance-reduction term in the recursion for \(\mathbb{E}\|e_{t+1}\|_F^2\). This term tightens the Lyapunov descent argument and yields the sharper \(T^{-1/3}\) dependence [2510.21800].

Conceptually, the significance of the theorem is not only that MARS-M improves the asymptotic rate, but also that it does so without abandoning the matrix-orthogonalization structure that makes Muon attractive for large matrices. The method therefore advances both the optimizer’s empirical design and its theoretical status.

## 5. Empirical results on language modeling and vision

The language-model experiments use GPT-2 style transformers at four scales—125M, 355M, 770M, and 1.5B—trained for 100k steps with context length 1024 on OpenWebText and FineWeb-Edu 100B. The setup uses 16 NVIDIA H800 GPUs for small models and 32 NVIDIA H800 GPUs for larger ones. Across these runs, MARS-M yields consistently lower training and validation losses than Moonlight, with the gap particularly visible in later training stages [2510.21800].

Downstream evaluation is performed on ARC, HellaSwag, MMLU, OpenBookQA, PIQA, SciQ, and WinoGrande, using 0-shot and 2-shot accuracy. For GPT-2 XL on FineWeb-Edu 100B in the 2-shot setting, Moonlight attains an average score of \(57.33\), while MARS-M reaches \(57.73\) with \(\gamma=0.01\) and \(57.70\) with \(\gamma=0.025\). Individual benchmark improvements include ARC-C \(40.87 \rightarrow 42.66\), HellaSwag \(55.80 \rightarrow 57.09\), SciQ \(91.90 \rightarrow 92.30\), and WinoGrande \(57.54 \rightarrow 58.09\) for the \(\gamma=0.025\) setting [2510.21800].

The computer-vision experiments use ResNet-18 on CIFAR-10 for 200 epochs with batch size 128. Here the paper compares Moonlight, exact MARS-M, and approximate MARS-M. Exact MARS-M achieves the lowest test loss and the highest test accuracy; approximate MARS-M remains better than Moonlight but slightly worse than the exact variant. This split is important because it isolates the effect of true same-batch variance reduction from the cheaper approximation used in language-model training [2510.21800].

An ablation over \(\gamma \in \{0.005, 0.01, 0.025, 0.05, 0.1\}\) on GPT-2 small shows that overly large \(\gamma\) hurts performance, while values in the range \(0.005\) to \(0.025\) behave similarly and robustly. This is consistent with the broader MARS observation that full-strength control-variate correction is too aggressive for deep-learning practice, whereas small, scaled corrections improve stability and speed [2510.21800].

## 6. Other documented meanings of “MARS-M”

Outside optimization, the exact string MARS-M also appears in sequential recommendation. In "MARS: Multi-rate Aggregation of Recency Signals for Sequential Recommendation across Sparse and Dense Regimes" [2606.03718], MARS-M is the Mamba-based instantiation of a timestamp-aware recency aggregation module. It is selected automatically for dense datasets according to the rule
\[
\text{backbone}(\mathcal{D})=
\begin{cases}
\text{MARS-T}, & \bar{L}(\mathcal{D})<50,\\
\text{MARS-M}, & \bar{L}(\mathcal{D})\ge 50,
\end{cases}
\]
and on ML-1M it reports HR@10 \(32.80\) versus \(31.79\) for SIGMA, together with \(42\%\) fewer MFLOPs [2606.03718]. In that literature, MARS-M therefore has nothing to do with variance reduction or Muon; it is a Mamba-based recommender.

A different situation holds in the multi-agent memory paper "MARS: Memory-Enhanced Agents with Reflective Self-improvement" [2503.19271]. There, the exact string does not appear explicitly. The closest interpretation is “MARS with its MemorySyntax module enabled,” corresponding to the “w memo” rows in the ablation table rather than an official named variant. The same pattern recurs in "MARS: Optimizing Dual-System Deep ReSearch via Multi-Agent Reinforcement Learning" [2510.04935], where the paper states that it never explicitly writes “MARS-M” and uses the label only as an informed extrapolation for the multi-agent reinforcement-learning component [2510.04935].

A common misconception is therefore to treat MARS-M as a single, portable acronym. The arXiv record does not support that reading. It supports, instead, a family of context-dependent usages: an explicitly named optimizer [2510.21800], an explicitly named Mamba recommender [2606.03718], and several inferred shorthands in other MARS frameworks [2503.19271]. Among these, the optimizer is the only case where “MARS-M” is itself the formal title of the method.

Source: https://www.emergentmind.com/topics/mars-m