Actor-Critic without Actor (ACA)

• ACA is a reinforcement learning framework that eliminates the separate actor network by using a noise-level critic to guide a stochastic diffusion chain for action generation.
• The method integrates a dual-loss training objective with a temporal-difference term and a noise-level regression to ensure smooth value estimation and effective denoising.
• Empirical results on MuJoCo tasks demonstrate ACA’s competitive performance and parameter efficiency compared to traditional actor-critic and diffusion-based methods.

Searching arXiv for the specified paper and closely related reinforcement-learning context. Actor-Critic without Actor (ACA) is a reinforcement-learning framework that removes the explicit actor network from actor-critic training and instead generates actions directly from the gradient field of a noise-level critic (Ki et al., 25 Sep 2025). In this formulation, a single critic $Q_\theta(s,a_t,t)$ both evaluates actions and guides a short diffusion chain that denoises from Gaussian noise toward high-value actions. The method is presented as a lightweight alternative to conventional actor-critic and diffusion-based policies, with the stated aims of reducing algorithmic and computational overhead, avoiding actor-induced policy lag, and preserving the capacity to represent diverse, multi-modal behaviors through stochastic denoising.

1. Conceptual position within actor-critic reinforcement learning

In standard off-policy actor-critic methods such as SAC, the learning system maintains a critic $Q_\theta(s,a)$ trained by temporal-difference targets and an actor $\pi_\phi(a\mid s)$ updated to choose actions with higher estimated $Q$-value. The exposition of ACA identifies three costs of this two-network setup: extra computation and memory, delicate hyperparameter tuning, and policy-lag, meaning that the actor can only slowly chase the critic’s updated value estimates (Ki et al., 25 Sep 2025).

ACA removes the explicit actor network entirely. Instead of learning a separate policy, it uses a single noise-level critic to both evaluate and directly generate actions by integrating its gradient field over a small diffusion chain. The stated consequences are a single network to train, immediate “on-the-fly” policy improvement, and preservation of multi-modality via stochastic denoising.

This placement is significant because ACA is framed simultaneously against two baselines. Relative to conventional actor-critic methods, it eliminates actor training. Relative to diffusion-based policies, it retains diffusion-style stochastic action synthesis but does so without a separate denoising policy network. The resulting formulation is explicitly described as combining simplicity with expressiveness.

2. Noise-level critic and state-action-noise representation

The central object in ACA is the noise-level critic

$$Q_\theta:\mathcal{S}\times\mathbb{R}^d\times\{0,\dots,T\}\longrightarrow\mathbb{R},$$

which takes a state $s$, a noised action $a_t$, and a discrete noise level $t\in\{0,1,\dots,T\}$, and outputs a scalar (Ki et al., 25 Sep 2025). The action corruption process is defined by

$$a_t \sim \mathcal{N}\!\bigl(a_0,(1-\bar\alpha_t)I\bigr), \qquad \bar\alpha_t=\prod_{i=1}^t(1-\beta_i),$$

with the shorthand $\sigma_t^2=1-\bar\alpha_t$.

The role of the critic varies with the noise level. For $Q_\theta(s,a)$0, $Q_\theta(s,a)$1 approximates the standard Bellman $Q_\theta(s,a)$2-value under the induced policy. For $Q_\theta(s,a)$3, $Q_\theta(s,a)$4 is trained to transport the $Q_\theta(s,a)$5 value back to noisy inputs. At generation time, $Q_\theta(s,a)$6 guides denoising toward a high-value clean action.

The implementation described for this critic is a multi-layer perceptron with 3 hidden layers of 256 units each, Mish activations, and a final linear head. The method maintains two copies, $Q_\theta(s,a)$7 and $Q_\theta(s,a)$8, connected by standard soft updates,

$Q_\theta(s,a)$9

A recurrent point in the ACA formulation is that action generation is not delegated to a separate policy class. The critic itself provides the local geometry of the action space through gradients with respect to $\pi_\phi(a\mid s)$0. This means that the critic is simultaneously a value estimator at $\pi_\phi(a\mid s)$1 and a denoising guide across $\pi_\phi(a\mid s)$2.

3. Training objective and critic-guided denoising

ACA uses a critic loss with two components: a temporal-difference term at noise level $\pi_\phi(a\mid s)$3, and a noisy-level regression term over $\pi_\phi(a\mid s)$4 with uniform schedule $\pi_\phi(a\mid s)$5 (Ki et al., 25 Sep 2025). The loss is given as

$\pi_\phi(a\mid s)$6

with

$\pi_\phi(a\mid s)$7

Under mild conditions, the minimizer of the noisy regression term satisfies

$\pi_\phi(a\mid s)$8

The text characterizes this as averaging over the terminal values of all clean actions that could have diffused to $\pi_\phi(a\mid s)$9. In the later theoretical discussion, this same property is described as making $Q$0 a smoothed version of the terminal critic $Q$1.

Action generation begins from

$Q$2

and then applies critic-guided denoising from $Q$3 down to $Q$4. In the pseudo-code, the update for a candidate action is

$Q$5

where $Q$6 is normalized as

$Q$7

The scalar $Q$8 is the guidance weight balancing exploitation and exploration. The exposition explicitly relates this update to classifier-guidance in diffusion, with the classifier gradient replaced by $Q$9. It also states that a first-order “Langevin” view is possible, although the diffusion chain was found to yield better multi-modal coverage. A clarification follows from this design: removing the actor does not remove stochasticity or diversity, because action generation still proceeds through a noisy denoising process rather than deterministic maximization.

4. Algorithmic workflow and implementation details

The full ACA procedure alternates between data collection and critic updates (Ki et al., 25 Sep 2025). During data collection, the system observes a state $$Q_\theta:\mathcal{S}\times\mathbb{R}^d\times\{0,\dots,T\}\longrightarrow\mathbb{R},$$0, generates $$Q_\theta:\mathcal{S}\times\mathbb{R}^d\times\{0,\dots,T\}\longrightarrow\mathbb{R},$$1 candidate actions by running $$Q_\theta:\mathcal{S}\times\mathbb{R}^d\times\{0,\dots,T\}\longrightarrow\mathbb{R},$$2 critic-guided denoising chains from Gaussian noise, and selects the final action

$$Q_\theta:\mathcal{S}\times\mathbb{R}^d\times\{0,\dots,T\}\longrightarrow\mathbb{R},$$3

The resulting transition $$Q_\theta:\mathcal{S}\times\mathbb{R}^d\times\{0,\dots,T\}\longrightarrow\mathbb{R},$$4 is stored in the replay buffer.

During learning, minibatches $$Q_\theta:\mathcal{S}\times\mathbb{R}^d\times\{0,\dots,T\}\longrightarrow\mathbb{R},$$5 are sampled from the replay buffer. For each sample, the next action $$Q_\theta:\mathcal{S}\times\mathbb{R}^d\times\{0,\dots,T\}\longrightarrow\mathbb{R},$$6 is generated by one diffusion chain, and the target is computed as

$$Q_\theta:\mathcal{S}\times\mathbb{R}^d\times\{0,\dots,T\}\longrightarrow\mathbb{R},$$7

A noise level $$Q_\theta:\mathcal{S}\times\mathbb{R}^d\times\{0,\dots,T\}\longrightarrow\mathbb{R},$$8 is sampled, Gaussian noise is injected into the stored action, and the critic is trained with the batch loss

$$Q_\theta:\mathcal{S}\times\mathbb{R}^d\times\{0,\dots,T\}\longrightarrow\mathbb{R},$$9

Several implementation choices are highlighted as key components. Batch action sampling generates $s$0 candidate denoised actions and picks the one with highest $s$1 to reduce noise variance. Gradient normalization divides $s$2 by $s$3 to stabilize the denoising chain. The noise schedule is $s$4, and $s$5 are chosen as a cosine or linear schedule; the text notes that in practice 20 steps works well.

The implementation summary further specifies: replay buffer capacity $s$6, warmup $s$7, batch $s$8, discount $s$9, soft-update $a_t$0, diffusion steps $a_t$1 by default, guidance weight $a_t$2 (tuned), candidate actions per state $a_t$3, noise scale $a_t$4, critic learning rate $a_t$5, $a_t$6-LR $a_t$7, target entropy $a_t$8, and gradient clipping by normalizing $a_t$9 to unit norm. The same section also states the use of two critics (double-Q) to reduce overestimation.

5. Theoretical characterization and convergence intuition

The formal theoretical statement presented for ACA is Proposition 1, termed noise-level consistency (Ki et al., 25 Sep 2025). Under exact minimization of the noisy regression term, for each fixed state $t\in\{0,1,\dots,T\}$0 and noise level $t\in\{0,1,\dots,T\}$1,

$t\in\{0,1,\dots,T\}$2

where $t\in\{0,1,\dots,T\}$3 is the forward diffusion posterior. The exposition interprets this as ensuring that $t\in\{0,1,\dots,T\}$4 is a smoothed version of the terminal critic $t\in\{0,1,\dots,T\}$5, so that $t\in\{0,1,\dots,T\}$6 remains well-conditioned even when $t\in\{0,1,\dots,T\}$7 is heavily corrupted by noise.

The convergence discussion is framed as intuition rather than a formal theorem. Because there is no actor network, there is no separate policy-gradient loop; policy improvement is performed immediately via the denoising chain using the same network that is being trained. Under standard contraction properties of the Bellman operator and small diffusion step sizes, ACA is said to inherit the usual TD convergence guarantees for the critic, plus empirical stability from the noise-level regularization.

A common misunderstanding would be to treat ACA as value-based action selection without policy structure. The formulation instead embeds policy improvement in a stochastic diffusion process conditioned by critic gradients. Another misunderstanding would be to equate actor removal with loss of multi-modality; the method explicitly claims preservation of multi-modal action sampling through stochastic denoising and states that the diffusion chain yielded better multi-modal coverage than the alternative first-order “Langevin” view.

The exposition also notes that a formal convergence theorem would parallel that of SAC except that the soft-policy is sampled via diffusion guidance rather than a trained Gaussian policy. This is presented as a prospective theoretical alignment rather than a completed theorem.

6. Empirical profile, ablations, and comparative claims

On online MuJoCo with 1 M steps, ACA is reported to outperform or match SAC, QSM, DIPO, DACER, QVPO, and SDAC on 10 tasks: Ant, HalfCheetah, Hopper, Walker2d, Humanoid, Swimmer, Pusher, Reacher, InvertedPendulum, and InvertedDoublePendulum (Ki et al., 25 Sep 2025). At 100 k steps, ACA’s mean returns exceed all baselines on 5 of 6 standard tasks. The paper summary characterizes the resulting learning curves as more favorable while describing overall performance as competitive with both standard actor-critic and state-of-the-art diffusion-based methods.

The parameter-efficiency claim is explicit: ACA uses a single critic, approximately $t\in\{0,1,\dots,T\}$8k parameters, which is $t\in\{0,1,\dots,T\}$9 the parameters of SAC ($$a_t \sim \mathcal{N}\!\bigl(a_0,(1-\bar\alpha_t)I\bigr), \qquad \bar\alpha_t=\prod_{i=1}^t(1-\beta_i),$$0k) and approximately $$a_t \sim \mathcal{N}\!\bigl(a_0,(1-\bar\alpha_t)I\bigr), \qquad \bar\alpha_t=\prod_{i=1}^t(1-\beta_i),$$1 those of diffusion-actor methods. The significance assigned to this comparison is that ACA reduces the parameter and tuning burden associated with maintaining a separate actor, while still performing critic-guided policy improvement.

In the Offline$$a_t \sim \mathcal{N}\!\bigl(a_0,(1-\bar\alpha_t)I\bigr), \qquad \bar\alpha_t=\prod_{i=1}^t(1-\beta_i),$$2Online HalfCheetah O$$a_t \sim \mathcal{N}\!\bigl(a_0,(1-\bar\alpha_t)I\bigr), \qquad \bar\alpha_t=\prod_{i=1}^t(1-\beta_i),$$3O setting, ACA is reported to match or surpass CQL, IQL, Cal-QL, WSRL, and RLPD without any offline pre-training and with only a double-Q critic, rather than large ensembles. This comparison is presented as evidence that ACA’s critic-only architecture is not restricted to standard online control benchmarks.

The ablation summary isolates two principal controls. For the guidance weight $$a_t \sim \mathcal{N}\!\bigl(a_0,(1-\bar\alpha_t)I\bigr), \qquad \bar\alpha_t=\prod_{i=1}^t(1-\beta_i),$$4, the text states: too small $$a_t \sim \mathcal{N}\!\bigl(a_0,(1-\bar\alpha_t)I\bigr), \qquad \bar\alpha_t=\prod_{i=1}^t(1-\beta_i),$$5 over-explore; too large $$a_t \sim \mathcal{N}\!\bigl(a_0,(1-\bar\alpha_t)I\bigr), \qquad \bar\alpha_t=\prod_{i=1}^t(1-\beta_i),$$6 greedy; sweet-spot $$a_t \sim \mathcal{N}\!\bigl(a_0,(1-\bar\alpha_t)I\bigr), \qquad \bar\alpha_t=\prod_{i=1}^t(1-\beta_i),$$7. For the number of denoising steps $$a_t \sim \mathcal{N}\!\bigl(a_0,(1-\bar\alpha_t)I\bigr), \qquad \bar\alpha_t=\prod_{i=1}^t(1-\beta_i),$$8, it states that $$a_t \sim \mathcal{N}\!\bigl(a_0,(1-\bar\alpha_t)I\bigr), \qquad \bar\alpha_t=\prod_{i=1}^t(1-\beta_i),$$9 balances performance and computational cost. Together with batch candidate selection and gradient normalization, these observations define the practical regime in which the method is reported to work well.

The broader interpretation suggested by these results is that ACA aims to preserve multi-modal action generation and immediate policy improvement while requiring fewer parameters and fewer hyperparameters to tune than conventional actor-critic or diffusion-actor approaches. Within the presented account, that combination constitutes the method’s principal contribution.