---
title: Equivariant Soft Actor-Critic (Equi-SAC)
url: https://www.emergentmind.com/topics/equivariant-soft-actor-critic-equi-sac
type: topic
---

# Equivariant Soft Actor-Critic (Equi-SAC)

Equivariant Soft Actor-Critic (Equi-SAC) is a reinforcement learning (RL) algorithm that enforces exact SO(2)-equivariance in both actor and critic networks via steerable group convolutions. By embedding symmetry properties at the algorithmic and architectural levels, Equi-SAC achieves substantial improvements in sample efficiency, particularly in robotic manipulation tasks characterized by underlying rotational symmetries [2203.04439].

## 1. Theoretical Foundations

Equi-SAC is grounded in the mathematical theory of group actions and equivariant representations. Consider the group $G = SO(2)$, the group of planar rotations, approximated in practice by its discrete cyclic subgroup $C_n = \{\mathrm{Rot}_\theta \mid \theta = 2\pi i/n,\; i=0,\dots,n-1\}$.

**States** are encoded as $m$-channel images:
\[
\mathcal{F}_s : \mathbb{R}^2 \to \mathbb{R}^m,
\]
with group action $g \in C_n$ implemented as rotational shifts in pixel space,
\[
(g\mathcal{F}_s)(x, y) = \mathcal{F}_s\bigl(g^{-1}(x, y)\bigr),
\]
where channel values follow the trivial representation $\rho_0$.

**Actions** $a \in A \subset \mathbb{R}^k$ are partitioned as
\[
a = (a_{\text{equiv}},\, a_{\text{inv}}),
\]
with $a_{\text{equiv}} \in \mathbb{R}^2$ transforming under the standard representation $\rho_1$—that is,
\[
g \cdot a = (\rho_1(g) a_{\text{equiv}},\, a_{\text{inv}}).
\]

Under these definitions, the optimal value and policy functions exhibit strict group-theoretic structure:
\[
Q^*(g s,\, g a) = Q^*(s, a),\qquad
\pi^*(g s) = g(\pi^*(s)).
\]

Equi-SAC enforces these properties by construction:
- **Critic invariance**: $Q(e(g \mathcal{F}_s),\, g a) = Q(e(\mathcal{F}_s),\, a)$.
- **Actor equivariance**: $\pi(g \mathcal{F}_s) = g(\pi(\mathcal{F}_s))$.

## 2. Network Architectures

Equivariant layers in Equi-SAC are implemented using group convolutions satisfying
\[
K(\rho_1(g)\, v) = \rho_{\text{out}}(g)\, K(v)\, \rho_{\text{in}}(g)^{-1}.
\]
These layers are instantiated with the E2CNN library and respect rotational symmetries $C_n$.

### Actor Network

- **Input:** $128 \times 128$ depth image ($2$-channel trivial representation).
- **Core:** $8$ steerable convolutional layers ($3 \times 3$ kernels, ReLU activations), equivariant under $C_8$.
- **Output:** $1 \times 1$ tensor comprising:
    - One $\rho_1$ vector (two degrees of freedom) for $a_{\text{equiv}}$.
    - Eight $\rho_0$ scalars for $(a_{\text{inv}},\, \sigma)$.

The steerable basis enables a single convolution to handle all rotated filter versions.

### Critic Network

- **Encoder:** $7$ steerable convolutional layers mapping $\rho_0 \to \rho_{\text{reg}}$, terminate in $1 \times 1$ features ($\mathbb{R}^{n \times 64}$).
- **Action concatenation:** Actor output ($\rho_1$ vector $+$ $3$ $\rho_0$ scalars) is appended to encoder output.
- **Heads:** Two $Q$-functions, each a stack of two convolutional layers (regular to trivial), concluding with max-pooling over group channels to overcome Schur’s Lemma constraints.
- **Output:** Two scalar $Q$ values $Q_1,\, Q_2$.

#### Architecture Sketch

\[
\begin{aligned}
\text{Actor:} \quad & \mathcal{F}_s \xrightarrow[\rho_0]{\text{C-SteerConv} \rightarrow \ldots \rightarrow \text{ReLU}}
\underbrace{
    \left[
        \underbrace{\rho_1}_{a_{xy}}
        \oplus
        \underbrace{\rho_0^8}_{a_{\lambda}, a_z, a_{\theta}, \sigma}
    \right]
}_{1\times1}
= \pi(\mathcal{F}_s). \\
\text{Critic:} \quad &
\mathcal{F}_s
\xrightarrow[\rho_0]{\text{SteerConv}}
\ldots
\xrightarrow[\rho_{\text{reg}}]{\text{1$\times$1 conv}}
e(\mathcal{F}_s) \in (\mathbb{R}^n)^{64}
\rightarrow (e(\mathcal{F}_s) \oplus a)
\rightarrow \text{SteerConv}
\rightarrow \text{max}_{g \in G}
\xrightarrow{\rho_0} Q(e(\mathcal{F}_s), a).
\end{aligned}
\]

## 3. Algorithm and Training Protocol

The Equi-SAC learning process follows an adaptation of the standard Soft Actor-Critic (SAC) framework, with explicit equivariant/invariant losses and update rules:

1. **Replay Buffer:** $\epsilon$-greedy replay buffer $R$, with optional demonstration data.
2. **Action Sampling:** Actor ($\pi_\phi$) outputs mean and standard deviation; action generated via reparameterization trick. Equivariant components are rotated into a base frame before execution.
3. **Critic Update:** The target value is computed using soft target networks and incorporates the log probability under the policy, maintaining critic invariance under $SO(2)$.
4. **Actor Update:** The loss involves both entropy maximization and critic evaluation, enforcing actor equivariance through the network structure.
5. **Temperature Update:** Entropy temperature $\alpha$ is optionally updated to match target entropy.
6. **Target Network:** Soft update ($\tau = 10^{-2}$) is used for critic target networks.

All convolutional operations inherit $SO(2)$-equivariance, ensuring that both actor and critic properly respect symmetry constraints throughout optimization.

### Pseudocode (abridged)

```
Given: replay buffer R, equivariant networks π_φ, Q_θ1/Q_θ2, targets θ̄1/θ̄2, α

For each env step:
    - Observe s, encode as image 𝓕_s
    - Sample action: (μ,σ) ← π_φ(𝓕_s); a = μ + σ⊙ε (ε ∼ N(0,I))
    - Rotate a_equiv, append a_inv; execute a
    - Observe (r, s'), store in R

On update:
    - For k=1..K batches:
        - Critic update: y = r + γ[min_j Q_{θ̄_j}(e(𝓕_{s'}),a') - α log π_φ(a'|𝓕_{s'})]
        - θ_i ← θ_i - η∇L_Q
        - Actor update: ã~π_φ(𝓕_s); φ ← φ - η∇L_π
        - α update (optional)
        - θ̄_j ← τ θ_j + (1−τ) θ̄_j
```

## 4. Empirical Evaluation

### Benchmarks

Equi-SAC was benchmarked on six robotic visual-control tasks:
- **Simple:** Block Pulling, Object Picking, Drawer Opening
- **Hard:** Block Stacking, House Building, Corner Picking

Each environment used $128 \times 128$ depth images as state observations, continuous action spaces $A \subset \mathbb{R}^5$, and sparse rewards (+1 for task success).

### Sample Efficiency

Steps required to achieve 80% success rate (mean across 4 seeds):

| Task           | Equi-SAC | CNN-SAC | DrQ   | RAD   | FERM  |
|----------------|----------|---------|-------|-------|-------|
| Block Pull     | 45k      | 180k    | 150k  | 170k  | 130k  |
| Object Pick    | 80k      | ×       | ×     | ×     | ×     |
| Drawer Open    | 90k      | ×       | ×     | ×     | ×     |

($\times$ indicates failure to solve within 300k steps.)

### Ablation Study

Effect of equivariant actors and critics (final reward after 200k steps):

|                | EqActor+EqCrit | EqActor+CNNCrit | CNNActor+EqCrit |
|----------------|----------------|-----------------|-----------------|
| Pull           | **0.97**       | 0.82            | 0.75            |
| Pick           | **0.92**       | 0.70            | 0.65            |
| Open           | **0.95**       | 0.78            | 0.72            |

$C_8$ symmetry (eightfold rotation) consistently outperformed $C_4$ (fourfold) in 5 of 6 domains.

## 5. Implementation and Practical Considerations

Key engineering and hyperparameter guidelines:

- **Steerable CNNs:** Use E2CNN or similar libraries for group convolutions. Specify input/output representations ($\rho_{\text{in}}, \rho_{\text{out}}$) to enable weight sharing across formal group actions.
- **Critic Pooling:** Apply group-max pooling before the final scalar output to avoid overconstraint (Schur’s Lemma).
- **Action Augmentation:** When employing buffer augmentations, rotate $a_{\text{equiv}}$ by the same SO(2) element as the state.
- **Critical Hyperparameters:**
    - Soft update rate $\tau = 10^{-2}$
    - Actor and critic learning rates: $10^{-3}$ (Adam optimizer)
    - Entropy temperature initialization: $\alpha = 10^{-2}$, target entropy $-5$
    - Batch size: $64$
    - Replay buffer capacity: $10^5$

This structure ensures exact $SO(2)$-equivariance, resulting in marked improvements in both sample efficiency and final policy performance on visual-control benchmarks.

## 6. Significance and Context

Equi-SAC demonstrates that explicitly modeling group symmetry within RL architectures can lead to substantial empirical gains in data efficiency and task performance, particularly for domains where physical laws or robot/environment geometry induce SO(2)-invariant MDPs. These results provide a basis for adopting symmetry-preserving architectures in broader robot learning applications and for extending equivariant RL principles to other symmetry groups beyond SO(2) [2203.04439].

Source: https://www.emergentmind.com/topics/equivariant-soft-actor-critic-equi-sac