Revisiting Action Factorization for Complex Action Spaces

Abstract: Many real-world control problems involve hybrid discrete-continuous action spaces. For example, steering and signaling in autonomous driving, and aiming and firing in robotics or video-games. Despite real-world hybrid factorization and reinforcement learning framework support for complex action spaces (e.g., Gymnasium, PettingZoo, TorchRL, SeedRL, Mujoco, etc), the default environments within those frameworks often implement uniform action space configurations (LunarLander, Walker2D, Cheetah, SMAC, SUMO, Ant, Atari). Landmark hybrid-action benchmarks (RoboCup 2D HFO, SC2LE, Platform, CARLA, etc) are mostly heavyweight or archival implementations originating from papers which test one or a small number of competing factorization methods on one kind of control. This article provides a cross-sectional study of factorization methods [independent networks, shared encoder, VDN, QPLEX, Joint, Auto-Regressive] on each of three families of algorithms [PPO, SAC, DQN] across three action spaces [discretized, hybrid, continuous] over four lightweight environments [Platform, hybrid-LunarLander, Hybrid-Shoot, CoopPush]. Accounting for some invalid pairings such as joint-continuous, we are left with 220 configurations to analyze each method. We provide two new C++ parallel gymnasium and petting-zoo compliant environments [CoopPush, Hybrid-Shoot] to isolate particular challenges such as state-dependent inter-action dependence. Finally, we introduce VDN-PPO and PPO-MIX which use a branching critic to assign credit to multi-headed PPO. These variants out-perform all other tested PPO factorizations. Our results suggest that branching dueling architectures balance compute and performance most effectively, with Auto-Regressive actions reaching the highest performance overall and native continuous SAC outperforming discrete and hybrid algorithms, albiet both at increased computational cost.

• The paper introduces a unified evaluation of factorization methods in RL, demonstrating how branching dueling critics achieve variance reduction.
• It systematically compares techniques like independent, shared encoder, auto-regressive, and joint actions across 220+ configurations.
• Empirical results show that branching architectures excel in performance and efficiency in complex, hybrid action spaces.

Revisiting Action Factorization for Complex Action Spaces

Motivations and Scope

"Revisiting Action Factorization for Complex Action Spaces" (2606.26574) addresses the fragmentation of factorization strategies for reinforcement learning (RL) in high-dimensional, mixed discrete-continuous action spaces. RL algorithms often default to uniform action spaces in widely used environments, while real-world settings—autonomous driving, mixed-mode robotics—call for hybrid actions and temporally or contextually dependent sub-action coordination. Prior benchmarks either focus narrowly on a single factorization or test specific configurations on heavyweight or legacy environments, with limited practical guidance for cross-family, cross-space settings.

This study implements a unified comparison of factorization paradigms—Independent, Shared Encoder (with subtypes including VDN and QPLEX), Joint, Auto-Regressive—across three algorithm families (PPO, SAC, DQN), each evaluated in discretized, hybrid, and continuous spaces. Evaluation covers 220+ algorithm-factorization-environment combinations, leveraging four lightweight environments (including two new: CoopPush, Hybrid-Shoot) exposing tunable action dependence. Notably, two new PPO variants with branching critics (VDN-PPO, PPO-MIX) perform explicit per-head credit assignment, targeting multi-action variance reduction. The paper's analytic arc is grounded in both numerical comparison and theoretical support for importance-weighted variance reduction.

Factorization Strategies: Architecture and Credit Assignment

Structural Paradigms

Factorizing complex action spaces is crucial for tractable RL. The main strategies are:

• Independent Networks: Each action head is modeled by its own policy, updated by a shared global training signal. This offers scalability but induces local equilibria and credit assignment ambiguity.
• Shared Encoder: A single state representation feeds several action heads. Three modes:
  • No Mixing: All heads get the reward signal directly, treating others' actions as noise.
  • VDN-style: State-value and per-head advantages are additively composed, strictly enforcing Individual-Global-Max (IGM).
  • QPLEX-style: Per-head advantages are monotonically mixed, allowing for nonlinear credit transfer.
• Branching Dueling Architectures: Both SAC-BDQ (Hybrid-SAC) and BDQ-DQN exploit branching dueling critics, assigning advantages per head conditioned on the state and, in SAC's case, continuous actions.
• Auto-Regressive: Actions are sampled sequentially, each conditioning on prior heads, enabling modeling of arbitrary dependencies but incurring sequential inference latency and elevated variance.
• Joint Actions: All sub-action dimensions are treated as a single Cartesian product, suitable for small action spaces but imposing exponential computational overhead.

Branching Critics and Importance Weighting

A fundamental innovation lies in explicit per-head credit assignment via branching dueling critics. The critic is trained with a $V$-only target, fitting the sum of per-head advantages to the TD residual. In VDN-PPO, each head receives an importance-weighted GAE, with per-head weights annealed by the advantage range. This guarantees:

• Unbiased global policy gradient.
• Redistribution of variance: variance budget is concentrated in high-importance heads, suppressed elsewhere.
• Identifiability via max-centering, avoiding mixer-induced gain inflation.

PPO-MIX employs QPLEX-style monotonic mixing, trading representational capacity for theoretical guarantees, with per-head weights derived from first-order Taylor expansion of the mixer's output with respect to the observed action.

Benchmark Environments

The suite comprises four environments tuned for action dependence:

• Contextual-Decoupler: Minimal bandit environment, isolates per-head credit assignment.
• CoopPush: Novel physics-based multi-particle domain with tunable action dependence; tests for boulder pushing cooperation.
• Hybrid-Shoot: Parameterized targeting, examines precision and cross-head dependence.
• Platform: Classic parameterized platformer, demonstrates discrete action selection with continuous parameterization.

  Figure 1: Visual depiction of LunarLander, CoopPush, Hybrid-Shoot, and Platform environments illustrating varied hybrid and dependent action spaces.

Empirical Analysis: Impact of Factorization

Contextual-Decoupler: Isolating Credit Assignment

Vectorized reward curves demonstrate that branching dueling architectures, particularly SAC-BDQ, instantly solve the task. DQN variants plateau due to exploration noise; VDN-PPO and PPO-MIX approach optimality, while unfactored scalar value PPO underperforms. SAC-Concat's performance is heavily entropy-sensitive due to joint relaxation.

Figure 2: Comparative reward progression for various algorithm/critic family pairings on Contextual-Decoupler; branching dueling critics converge fastest.

Variance reduction effects are highly significant: importance weighting suppresses inactive head variance to 83–86% of the active head, yielding a robust but second-order improvement in final reward (~4%). Both exact range-based and first-order estimators are statistically equivalent for learning dynamics, but only the exact weights reliably track ground-truth head responsibility.

Figure 3: Correlation and accuracy curves for range-based head importance weights versus true active head, confirming theoretical tracking and empirical variance reduction.

Aggregate Results: Global Comparison

Mean normalized results across all environments reveal:

• Branching dueling / VDN yields strongest compute-to-performance tradeoff.
• Auto-regressive factorization attains highest performance at increased computational cost; sequential bottlenecks are present.
• Discrete SAC underperforms in large hybrid/discrete spaces, aligning with entropy misalignment.

  Figure 4: Algorithm and action type aggregation highlighting the dominance of branching dueling and auto-regressive methods across the spectrum.

SAC achieves optimal performance in pure continuous regimes, but the hybrid implementations require careful separation of temperature coefficients to avoid destructive entropy interference.

Environment-Specific Observations

The shoot environment exposes SAC's failures in large discrete spaces due to misaligned entropy objectives—optimal policies necessarily converge to lower entropy as targets are eliminated, conflicting with soft actor-critic exploration rewards.

Figure 5: Independent Shoot environment SAC results; continuous SAC reaches high scores, discrete actions lag due to entropy misalignment.

Practical and Theoretical Implications

Model Selection Guidance

• Branching dueling (VDN) is the default recommendation for discrete or hybrid action spaces, offering low implementation and compute overhead.
• Auto-regressive actions excel when computational resources permit, scaling with the number (not the space) of actions.
• QPLEX/PPO-MIX is only advantageous for strongly coupled actions or continuous spaces; otherwise, scaling issues and implementational pitfalls dominate.
• For hybrid SAC, careful entropy separation is mandatory; performance is environment-dependent.
• Joint action factorization is prohibitively expensive for high-dimensional spaces, despite efficient parallel matrix multiplication on GPU.

Theoretical Guarantees, Limitations, and Future Directions

Importance-weighted credit assignment theoretically guarantees unbiasedness, variance reduction, and identifiability in multicritic architectures. Empirical results confirm order-of-magnitude gains from factorization, with variance redistribution as a secondary benefit. PPO-MIX improves representational capacity but introduces estimator covariance, sacrificing some theoretical guarantees.

Research directions involve:

• Scaling factorization strategies to massive action spaces and difficult exploration environments.
• Investigating equilibrium selection and symmetry-induced exploration hardness.
• Studying the interaction of factorization with asynchronous distributed frameworks, e.g., IMPALA and PQN.

Conclusion

This paper systematically analyzes a broad range of action space factorizations across algorithm families and environment complexities. Numerically, branching dueling and auto-regressive architectures provide the best performance in terms of sample efficiency, credit assignment, and scalability. Theoretically, importance-weighted variance reduction for branching critics is validated. The recommendations are clear for practitioners: default to VDN-type branching in fully observable settings, consider auto-regressive for performance if latency is acceptable, and avoid joint and monotonic mixing for large action spaces. Further research into scaling and distributed RL is encouraged.