Score-Matched Actor-Critic (SMAC)

• Score-Matched Actor-Critic (SMAC) is a reinforcement learning method that aligns the policy's score with the critic’s action-gradient for efficient, optimal updates.
• It employs a diffusion-based actor in continuous control and regularizes the critic to ensure robust offline-to-online transfer without initial performance drops.
• Empirical results demonstrate that SMAC improves sample efficiency and exploration while smoothing the transition from offline training to effective online fine-tuning.

Score-Matched Actor-Critic (SMAC) denotes actor-critic methods built around a first-order correspondence between a policy score and the action-gradient of a critic. In the diffusion-policy formulation of "Learning a Diffusion Model Policy from Rewards via Q-Score Matching" (Psenka et al., 2023), the method is also called Q-Score Matching (QSM) and is developed for off-policy reinforcement learning in continuous action spaces. In "SMAC: Score-Matched Actor-Critics for Robust Offline-to-Online Transfer" (Lara et al., 19 Feb 2026), the same acronym denotes an offline reinforcement learning method that regularizes the critic so that offline optima transfer to online value-based fine-tuning without an initial drop in performance. Across both uses, the core premise is that actor-critic compatibility can be expressed through score-gradient alignment rather than through a simple behavior cloning term or unrestricted gradient-based fine-tuning.

1. Naming, scope, and problem setting

The term SMAC is used for two related constructions. The 2023 formulation addresses off-policy reinforcement learning with a diffusion-model actor in continuous domains, and the 2026 formulation addresses robust offline-to-online transfer in offline reinforcement learning. Both works center the relation between a policy score and $\nabla_a Q$, but they instantiate that relation differently (Psenka et al., 2023, Lara et al., 19 Feb 2026).

Work	Setting	Core mechanism
(Psenka et al., 2023)	Off-policy RL with a diffusion-model policy	Match the denoising score network to $\nabla_a Q_\phi(s,a)$
(Lara et al., 19 Feb 2026)	Offline RL for offline-to-online transfer	Regularize $\nabla_a Q_\phi(s,a)$ to match a scaled dataset score estimate

In the diffusion-policy setting, the motivation is that previous works with diffusion actors fail to exploit the score-based structure of diffusion models and instead utilize a simple behavior cloning term to train the actor, limiting their ability in the actor-critic setting. In the offline-to-online transfer setting, the motivation is different: modern offline RL methods can learn performant actor-critics, but fine-tuning them online with value-based RL algorithms typically causes immediate drops in performance. The 2026 work provides evidence consistent with the hypothesis that offline maxima for prior algorithms and online maxima are separated by low-performance valleys that gradient-based fine-tuning traverses (Lara et al., 19 Feb 2026).

2. Diffusion-policy formulation and denoising score networks

In the Q-Score Matching formulation, the stochastic policy $\pi_\theta(a|s)$ is represented in continuous action space $\mathcal A=\mathbb R^d$ as a $K$-step variance-preserving diffusion model conditioned on state $s$ (Psenka et al., 2023). The forward noising schedule is fixed as $\beta_1,\ldots,\beta_K\in(0,1)$, with transitions

$q(a^1|s)=\mathcal N(a^1;0,I),$

and

$$q(a^t|a^{t-1},s)=\mathcal N\!\bigl(a^t;\sqrt{1-\beta_t}\,a^{t-1},\beta_t I\bigr),\qquad t=2,\ldots,K.$$

A denoising score network is then learned as

$\nabla_a Q_\phi(s,a)$0

The final action $\nabla_a Q_\phi(s,a)$1 is sampled by running the standard reverse SDE or its discretization, for example Langevin dynamics or ancestral sampling. In this construction, the score of the marginal policy with respect to $\nabla_a Q_\phi(s,a)$2 is exactly

$\nabla_a Q_\phi(s,a)$3

This identity is the technical basis for replacing direct differentiation through the entire diffusion-sampling process with score matching against the critic. The paper emphasizes that the resulting converged policies are implicitly multi-modal and explorative in continuous domains, which is a direct consequence of using a diffusion actor rather than a unimodal Gaussian parameterization (Psenka et al., 2023).

3. First-order optimality and score-gradient alignment

The theoretical core of the 2023 formulation is a link between the structure of the policy score and the action-gradient of the Q-function. At the global optimum of the expected discounted return

$\nabla_a Q_\phi(s,a)$4

any maximizing policy $\nabla_a Q_\phi(s,a)$5 must satisfy, for all $\nabla_a Q_\phi(s,a)$6 in support where $\nabla_a Q_\phi(s,a)$7,

$\nabla_a Q_\phi(s,a)$8

The paper outlines the key derivation in the deterministic setting through the path-value

$\nabla_a Q_\phi(s,a)$9

and a perturbation argument on the actor’s vector field $\nabla_a Q_\phi(s,a)$0. A Taylor expansion around $\nabla_a Q_\phi(s,a)$1 shows that if the perturbed drift acquires a component along $\nabla_a Q_\phi(s,a)$2 that the original drift lacked, then the path value increases. Repeating this argument throughout state-action space yields the stationary condition that the actor score must be parallel to the action-gradient of the critic (Psenka et al., 2023).

The 2026 formulation states the corresponding identity in maximum-entropy reinforcement learning as

$\nabla_a Q_\phi(s,a)$3

and approximately enforces during training

$\nabla_a Q_\phi(s,a)$4

Taken together, these two formulations distinguish between a support-wise collinearity condition and an exact scaled equality. The 2023 paper further states that any local improvement step that better aligns $\nabla_a Q_\phi(s,a)$5 with $\nabla_a Q_\phi(s,a)$6 strictly increases the expected return $\nabla_a Q_\phi(s,a)$7, and that the only fixed points satisfy

$\nabla_a Q_\phi(s,a)$8

on the support of the policy. Under mild regularity, the resulting stationary policy coincides with the Boltzmann policy $\nabla_a Q_\phi(s,a)$9 in the continuous-time limit (Psenka et al., 2023).

4. Losses, updates, and implementation

In Q-Score Matching, the policy update is defined directly in terms of score matching against the critic. At each diffusion time-step $\pi_\theta(a|s)$0, the noised action is

$\pi_\theta(a|s)$1

and the actor objective is

$\pi_\theta(a|s)$2

with weighting factors $\pi_\theta(a|s)$3 that are often set to $\pi_\theta(a|s)$4. In practice, the method maintains a replay buffer $\pi_\theta(a|s)$5 of transitions $\pi_\theta(a|s)$6. The critic is updated by TD learning, for example in TD3 or SAC style, using target

$\pi_\theta(a|s)$7

where $\pi_\theta(a|s)$8 is sampled from the current diffusion actor, and minimizing

$\pi_\theta(a|s)$9

The actor update samples a minibatch $\mathcal A=\mathbb R^d$0, generates noised versions $\mathcal A=\mathbb R^d$1, computes $\mathcal A=\mathbb R^d$2 by automatic differentiation of the frozen critic, and minimizes

$\mathcal A=\mathbb R^d$3

All gradients in $\mathcal A=\mathbb R^d$4 flow only through the score network $\mathcal A=\mathbb R^d$5; the Q-gradient is treated as a fixed target. The paper’s pseudocode initializes critics $\mathcal A=\mathbb R^d$6, target critics $\mathcal A=\mathbb R^d$7, a score network $\mathcal A=\mathbb R^d$8, and a replay buffer, then alternates data collection, critic updates, actor score updates, and Polyak target updates (Psenka et al., 2023).

The 2026 SMAC formulation modifies only the offline phase of an otherwise standard actor-critic pipeline. It estimates the dataset score $\mathcal A=\mathbb R^d$9 via a return-conditioned diffusion model $K$0, introduces a small network $K$1 to scale that estimate, and defines the score-matching regularizer

$K$2

where $K$3 is a 50/50 mix of actions from $K$4 and uniform random actions. The full critic loss is

$K$5

while the actor remains unchanged from SAC with loss $K$6. In practice the method also uses target networks $K$7 and ensemble critics as in SAC. The reported offline pseudocode updates $K$8, $K$9, and $s$0 with the Muon optimizer, described as a spectral-norm-aware optimizer, and reports typical hyperparameters $s$1, $s$2, $s$3, $s$4, batch size $s$5, ensembles of $s$6 critics, and target $s$7 (Lara et al., 19 Feb 2026).

5. Empirical findings

The diffusion-policy Q-Score Matching paper benchmarks SMAC on six DeepMind Control Suite tasks, including Walker Walk and Cheetah Run, and compares it to TD3 and SAC. The reported findings are threefold: sample efficiency matches or outperforms SAC and TD3, especially in the low-data regime; the learned diffusion actor naturally represents multi-modal action distributions, with the paper citing two swing-up modes in cartpole as an example; and exploration improves because sampling from the full diffusion chain, with optional Gaussian noise, visits diverse actions that Gaussian actors cannot. Figure 1 in the paper shows cumulative returns versus environment steps, and Figure 2 illustrates policy samples for a toy swing-up (Psenka et al., 2023).

The offline-to-online transfer paper evaluates SMAC on six D4RL tasks and reports smooth transfer to Soft Actor-Critic and TD3 in 6/6 tasks. In 4/6 environments, it reduces regret by 34–58% over the best baseline. It also reports that all six SMAC pre-trained agents exhibit no initial drop when fine-tuned with SAC or TD3, whereas CalQL, IQL, and TD3+BC often suffer a 5–20% collapse before recovering (Lara et al., 19 Feb 2026).

Environment	Best baseline $s$8 SMAC	$s$9
door	120.0 $\beta_1,\ldots,\beta_K\in(0,1)$0 50.3	$\beta_1,\ldots,\beta_K\in(0,1)$1
hopper	293.7 $\beta_1,\ldots,\beta_K\in(0,1)$2 386.3	$\beta_1,\ldots,\beta_K\in(0,1)$3 (SMAC wins final)
kitchen	467.1 $\beta_1,\ldots,\beta_K\in(0,1)$4 131.4	$\beta_1,\ldots,\beta_K\in(0,1)$5
pen	8.0 $\beta_1,\ldots,\beta_K\in(0,1)$6 5.3	$\beta_1,\ldots,\beta_K\in(0,1)$7
relocate	98.1 $\beta_1,\ldots,\beta_K\in(0,1)$8 62.8	$\beta_1,\ldots,\beta_K\in(0,1)$9
walker2d	650.5 $q(a^1|s)=\mathcal N(a^1;0,I),$0 650.5	$q(a^1|s)=\mathcal N(a^1;0,I),$1 (tied)

These empirical results support two distinct claims. In the 2023 formulation, score matching enables expressive and exploratory continuous-control actors without differentiating through the entire diffusion sampler. In the 2026 formulation, score matching is used to reshape the offline objective so that first-order optimization can move from offline optima toward better online optima without traversing an initial low-reward region (Psenka et al., 2023, Lara et al., 19 Feb 2026).

6. Interpretation, misconceptions, and extensions

A central distinction between the two SMAC formulations is where score matching enters the actor-critic loop. In the diffusion-policy formulation, score matching is the actor update itself: the denoising score network is directly matched to the critic’s action-gradient. In the offline-to-online transfer formulation, the actor remains a Gaussian policy with the standard SAC actor loss, and score matching appears as a critic regularizer during offline pre-training (Psenka et al., 2023, Lara et al., 19 Feb 2026).

This distinction resolves a common naming ambiguity. SMAC is not a single fixed algorithmic template; it is a label applied to two methods that share a derivative-matching principle. It is also not equivalent to adding a behavior cloning term to actor-critic learning. The 2023 paper explicitly contrasts its method with prior diffusion-policy approaches that utilize a simple behavior cloning term to train the actor, and argues that this limits their ability in the actor-critic setting. Likewise, the diffusion-policy SMAC update does not require backpropagating through the entire diffusion-sampling process, because it only needs to differentiate through the denoising model rather than the entire diffusion model evaluation (Psenka et al., 2023).

The offline-to-online transfer work interprets performance valleys as a loss-landscape phenomenon: offline maxima for prior algorithms and online maxima lie in different basins, and straight-gradient fine-tuning paths cross low-reward regions. Its informal proposition is that, under SMAC regularization, there exists a path $q(a^1|s)=\mathcal N(a^1;0,I),$2 along which $q(a^1|s)=\mathcal N(a^1;0,I),$3 is monotonic. This suggests that the regularizer is intended not merely to improve offline return, but to alter the topology of the solution set relevant to subsequent online optimization (Lara et al., 19 Feb 2026).

The extensions explicitly proposed for the 2026 formulation include pairing the offline regularizer with SAC, TD3, TD3+BC, or AWR; applying it in vision-based or language-conditioned reinforcement learning by training a diffusion-based score model on mixed modalities; and combining the SMAC regularizer with alternative pessimism techniques such as CQL. A plausible implication is that score-gradient alignment can be regarded as a broader design principle for actor-critic compatibility across expressive policy classes and across offline, online, and transfer regimes (Lara et al., 19 Feb 2026).