IDAAC: Invariant Decoupled Advantage Actor-Critic

• IDAAC is a reinforcement learning framework that decouples actor and critic networks to prevent overfitting and improve generalization.
• It employs dual networks with an adversarial auxiliary loss to enforce invariant policy representations for stable advantage estimation.
• Empirical studies on Procgen and DeepMind Control Suite demonstrate improved test performance and a reduced training-test gap.

The Invariant Decoupled Advantage Actor-Critic (IDAAC) framework is a reinforcement learning methodology designed to improve generalization and sample efficiency by decoupling the actor and critic networks, and by enforcing invariance in policy representations with respect to task-irrelevant environmental factors. IDAAC derives its foundational motivation from both the need to avoid overfitting in complex, high-dimensional environments and from recent theoretical advances in variance reduction for actor-critic methods.

1. Motivations and Conceptual Foundations

IDAAC addresses two main shortcomings in standard deep actor-critic algorithms: overfitting due to shared representations and poor generalization in highly diverse environments. In typical implementations, the policy and value function share a feature encoder, forcing both objectives—action selection and value prediction—to rely on the same environmental representation. As highlighted in (Raileanu et al., 2021), this coupling leads to the policy overfitting on spurious or instance-specific cues necessary for value estimation but irrelevant or even detrimental for robust policy optimization.

The approach is theoretically supported by the projection theorem and control variate analysis in actor-critic methodologies (Benhamou, 2019). Viewing policy gradient estimation through the lens of control variate estimators and conditional expectation projections justifies both the decoupling and the design of invariant signal extraction. If the gradient estimator’s variance can be reduced by projecting high-variance signals onto an invariant subspace, then both sample efficiency and generalization are improved.

2. Architectural Decoupling and Advantage Estimation

IDAAC operationalizes its concept by deploying two physically separated neural networks: one for the policy and one for the value function. The policy network contains two output heads—one for action selection (e.g., distribution over actions) and one for generalized advantage prediction. The value network estimates state value independently and receives no gradients from the policy objectives.

Let $\phi_A$ denote the actor's representation and $\phi_C$ the critic's representation (as discussed in (Garcin et al., 8 Mar 2025)). The loss functions are:

Policy Network:

$$J_{IDAAC}(\theta) = J_\pi(\theta) + \alpha_s S_\pi(\theta) - \alpha_a L_A(\theta) - \alpha_i L_E(\theta),$$

where $J_\pi$ is the policy objective, $S_\pi$ is entropy bonus, $L_A$ is an advantage prediction loss, and $L_E$ is the invariance-inducing adversarial loss.

Value Network:

$$L_V(\phi) = \mathbb{E}_t \left[(V_\phi(s_t) - \hat{V}_t)^2\right],$$

with targets $\hat{V}_t$ computed via standard discounted returns or bootstrapped approaches.

This architectural separation enables the actor to specialize in action-relevant, invariant information and the critic in dynamics and value-specific features. Empirical studies in (Garcin et al., 8 Mar 2025) show that, when decoupled, $\phi_A$ encodes less task-instance mutual information and more invariant, policy-relevant cues, whereas $\phi_C$ absorbs detail-rich information necessary for stable advantage computation.

3. Invariance Regularization and Auxiliary Losses

To enforce invariance in the policy's representations, IDAAC introduces an adversarial auxiliary loss. The policy’s encoder $E_\theta(\cdot)$ is trained to produce representations that do not contain task-instance (level-specific) signals. This is achieved via an auxiliary discriminator $D_\psi(\cdot,\cdot)$, which is trained to predict temporal ordering between paired states, while the encoder is adversarially trained to prevent the discriminator’s success:

Discriminator Loss:

$$L_D(\psi) = -\log D_\psi(E_\theta(s_i), E_\theta(s_j)) - \log [1 - D_\psi(E_\theta(s_i), E_\theta(s_j))]$$

Encoder Loss:

$$L_E(\theta) = -\tfrac{1}{2} \log D_\psi(E_\theta(s_i), E_\theta(s_j)) - \tfrac{1}{2} \log [1 - D_\psi(E_\theta(s_i), E_\theta(s_j))]$$

By incorporating $L_E$ into the policy objective, the actor is constrained to encode only information invariant across episodes and levels.

4. Variance Reduction and Theoretical Underpinnings

IDAAC lattices the variance minimization principles established for actor-critic methods via control variate estimators and the projection theorem (Benhamou, 2019). The core mechanism is to treat advantage prediction and invariant auxiliary baselines as control variates:

Optimal Control Variate:

$\hat{m}' = \hat{m} - \alpha (\hat{t} - \tau),$

with $$\alpha^* = \mathrm{Cov}(\hat{m},\hat{t}) / \mathrm{Var}(\hat{t})$$

Multi-dimensional Setting:

$\hat{m}' = \hat{m} - \lambda^\top T$

where $T$ collects multiple invariant baselines and $$\lambda^* = \mathbb{E}[TT^\top]^{-1} \mathbb{E}[\hat{m}T]$$

These control variate relationships, when mapped onto IDAAC’s advantage calculation, suggest that multi-headed invariant advantage predictors can further reduce gradient variance, especially if the baselines are chosen to exploit underlying symmetries and correlations. Thus, the use of conditional expectations, optimal linear combination of baselines, and projection onto invariant subspaces together ensure minimal variance and unbiased gradient flow.

5. Empirical Generalization and Performance Benchmarks

IDAAC’s empirical results, reported on challenging benchmarks such as Procgen and DeepMind Control Suite, demonstrate superior generalization to unseen environments and robustness to spurious instance-specific distractors (Raileanu et al., 2021). Key results include:

• Aggregated test scores on Procgen test levels surpassing competitive baselines (mean score ≈ 163.7 ± 6.1).
• Robust generalization with reduced training-test gap compared to PPO, PPG, and UCB-DrAC.
• Outperformance on DeepMind Control Suite with visual distractors, attributed to adversarially enforced invariance.
• Implementation leverages PPO as backbone, replacing shared encoders with decoupled ResNet architectures, employing alternating update schedules, and careful hyperparameter tuning for invariance and advantage loss weights.

6. Comparison to Related Decoupling and Value-Improvement Methodologies

Recent studies in off-policy discrete-action actor-critic frameworks emphasize the importance of decoupling actor and critic regularization—for example, separating entropy terms in DSAC variants to achieve DQN-level performance (Asad et al., 11 Sep 2025). The IDAAC philosophy aligns with these themes, albeit from a variant angle: the objective is not only stabilization but specifically to avoid encoding task-instance noise in the policy network.

Complementary approaches such as value-improved actor-critic algorithms (Oren et al., 3 Jun 2024), which introduce extra “greedification” operators in the critic update while keeping actor improvements smooth, further highlight the significance of decoupled updates. Both these frameworks, and IDAAC, demonstrate that aggressive exploitation of critic estimates must be balanced with stable policy updates to ensure convergence and sample efficiency.

7. Significance and Ongoing Directions

IDAAC advances the field by providing a theoretically grounded mechanism for generalization in reinforcement learning, leveraging optimal variance reduction via control variates and projection theorems, and pioneering the empirical demonstration of decoupled, invariant architectures. Its implementation recommendations and empirical benchmarks establish a new standard for RL agents, particularly in environments with extensive procedural or distractor-driven variability.

Further research may investigate multi-dimensional invariant baselines, dynamic adjustment of invariance regularization based on environment diversity, integration of value-improvement trade-offs, and practical instantiation in large-scale RL benchmarks. The connection between mutual information in representation learning (Garcin et al., 8 Mar 2025) and invariance principles lays foundation for joint information-theoretic and control variate approaches in next-generation actor-critic algorithms.