Symmetry-Aware Training in RL

• Symmetry-Aware Training (SAT) is a technique that incorporates domain symmetries to augment data and regularize training, enhancing learning speed and policy robustness.
• It applies explicit symmetry transformations—such as mirroring trajectories—to double the training data and enforce invariance in policy and value functions.
• SAT significantly improves sample efficiency and generalization in reinforcement learning, especially in robotics and control domains with inherent physical symmetries.

Symmetry-Aware Training (SAT) is a principled approach in machine learning and reinforcement learning that injects explicit knowledge of domain symmetries—such as invariances under reflection, translation, or permutation—into the training process to improve sample efficiency, generalization, and convergence characteristics. SAT is founded on the observation that many systems, especially those inspired by biological or physical domains, possess inherent symmetries that can be leveraged to augment data and guide policy learning. Formal exploitation of these symmetries, rather than learning them implicitly, leads to substantial performance enhancements.

1. Fundamental Principles of Symmetry-Aware Training

Symmetry-aware training centers on the recognition and utilization of invariances present in the task structure or the agent’s environment. In domains such as locomotion, robotics, and image analysis, symmetries (e.g., reflection, rotation, translation) correspond to transformations under which reward functions and dynamics remain invariant.

In practical terms, SAT leverages explicit group-theoretic properties—particularly involutive transformations (such as mirroring about a sagittal plane)—to generate additional training samples or to regularize models. The primary principle is that the policy and value functions should remain consistent under these symmetry transformations: $$Q(s_{\text{mirrored}}, a_{\text{mirrored}}) = Q(s, a)$$ where $s_{\text{mirrored}}$ and $a_{\text{mirrored}}$ are obtained via deterministic symmetry operations (e.g., matrix reflections).

The approach generalizes classical data augmentation techniques from supervised learning (rotation, scale, translation invariances in image tasks) to RL and control domains where the agent, embodiment, and reward exhibit physical or structural symmetry.

2. Methodological Framework and Algorithmic Implementation

The core SAT methodology augments the agent’s experience by applying deterministic symmetry transformations to each observed trajectory. In the quadruped domain of the DeepMind control suite, this takes the form of reflectional symmetry about the midline (sagittal plane):

1. Trajectory Augmentation: For every trajectory composed of state-action pairs $(s_j, a_i)$, a mirrored trajectory $(s_j^{\text{mirrored}}, a_i^{\text{mirrored}})$ is generated. The mapping is defined by deterministic linear transformations (e.g., matrix multiplication with reflection matrices) applied to joint angles, velocities, and actuator commands.
2. Replay Buffer Expansion: The experience replay buffer is populated with both original and mirrored trajectories. This doubles the effective dataset size without additional environment interactions, making it particularly advantageous in data-limited regimes.
3. Policy and Value Updates: The policy (actor) and value (critic) functions are updated to maximize the likelihood and performance over both original and mirrored experience: $$\max_\theta \Bigg(\sum_{j,i} q_{ij} \log \pi_\theta(a_i|s_j) + \sum_{j,i} q_{ij} \log \pi_\theta(a_i^{\textrm{mirrored}}|s_j^{\textrm{mirrored}}) \Bigg)$$ where $q_{ij}$ are importance weights (from MPO; Maximum a Posteriori Policy Optimization), and $\pi_\theta$ is the parameterized policy.

For Q-function evaluation, both original and mirrored tuples are used, enforcing invariance: $$Q(s_{\text{mirrored}}, a_{\text{mirrored}}) = Q(s, a)$$ TD-targets and updates treat mirrored data identically.

This implementation requires only deterministic transformation functions and systematic augmentation logic, without modification to the underlying policy optimization algorithm.

3. Impact on Learning Speed, Sample Efficiency, and Generalization

Empirical evaluations conducted in the quadruped locomotion domain demonstrate unequivocal improvements when employing SAT. Under constrained data collection (simulating real-world robotics limits):

• Learning curves for symmetry-augmented agents exhibit faster convergence and reach higher cumulative reward in fewer environment interactions.
• Target performance metrics (e.g., distance walked, run stability) are achieved in a smaller number of episodes.
• SAT outperforms traditional training in both “walk” and “run” tasks, as shown in comparative plots.

The augmentation effectively regularizes learning, reducing overfitting to asymmetric or degenerate solutions and promoting robust generalization. The agent better exploits the underlying structure of the task, avoiding arbitrary, symmetry-breaking behavior.

A plausible implication is that in real robotics, where physical trial is costly or time-constrained, such augmentation yields substantial reductions in necessary experiment time and hardware wear.

4. Application Scope and Inductive Biases

SAT is directly applicable in any domain where symmetry is inherent in the reward function or task dynamics. This includes:

• Robotics: Multi-legged locomotion, manipulation tasks with interchangeable end-effectors.
• Modular agents: Systems whose components are structurally symmetric or configurable.
• Physical simulation: Fluid dynamics, molecular modeling, situations with spatial or parameter symmetries.

The method serves as an inductive bias, promoting solutions consistent with domain invariances and discouraging overfitting to non-essential detail or asymmetric policies. This bias can facilitate transfer learning; agents trained in symmetric tasks can adapt more efficiently to asymmetric extensions (e.g., adapting walk to uneven terrain).

5. Experimental Evidence and Quantitative Outcomes

Systematic experiments comparing standard (non-Augmented) and SAT-based approaches report that symmetry-aware models:

• Achieve better cumulative reward per episode in identical time budgets.
• Require fewer unique environment steps to solve tasks to the same specification.
• Demonstrate greater robustness to reduced batch sizes and sparser data.

Detailed analysis in Figure 1 and related tables of the referenced paper corroborate these claims both qualitatively and quantitatively.

6. Broader Significance in Machine Learning and Control

Symmetry-aware training provides a general framework for leveraging domain structure, moving beyond brute-force scaling or naïve architecture adaptation. Its principles parallel advances in equivariant neural networks and invariant data augmentation in other ML subfields.

In reinforcement learning for realistic control applications, SAT enables efficient exploitation of physics and geometry, offering sharply improved sample efficiency, faster policy convergence, and increased reliability of derived controllers.

SAT’s conceptual underpinnings suggest broader potential as a regularization and data efficiency tool in structured and scientific ML domains where symmetries are central yet often underutilized.

---

Summary Table: Core Elements of Symmetry-Aware Training in RL (Quadruped Domain)

Aspect	Method/Description	Outcome
Symmetry type	Reflectional (sagittal plane) in states/actions	Enables deterministic mirroring
Augmentation	Mirrored trajectory added for each sampled episode	Doubles data, no extra environment steps
Policy/value update	Both original and mirrored in MPO objective/Q-learning	Enforces reward invariance
Data regime impact	Most pronounced improvement in limited-data/small-batch setting	Faster, more stable learning
Inductive bias	Reduces overfitting to asymmetric policies; promotes regularity	Better generalization, stronger policies

---

SAT thus constitutes a practical, theoretically grounded paradigm for rapid policy learning and generalization in domains characterized by strong symmetries. Its algorithmic simplicity recommends it for widespread adoption in symmetric control and robotic tasks.