Delayed Homomorphic Reinforcement Learning for Environments with Delayed Feedback

Abstract: Reinforcement learning in real-world systems is often accompanied by delayed feedback, which breaks the Markov assumption and impedes both learning and control. Canonical state augmentation approaches cause the state-space explosion, which introduces a severe sample-complexity burden. Despite recent progress, the state-of-the-art augmentation-based baselines remain incomplete: they either predominantly reduce the burden on the critic or adopt non-unified treatments for the actor and critic. To provide a structured and sample-efficient solution, we propose delayed homomorphic reinforcement learning (DHRL), a framework grounded in MDP homomorphisms that collapses belief-equivalent augmented states and enables efficient policy learning on the resulting abstract MDP without loss of optimality. We provide theoretical analyses of state-space compression bounds and sample complexity, and introduce a practical algorithm. Experiments on continuous control tasks in MuJoCo benchmark confirm that our algorithm outperforms strong augmentation-based baselines, particularly under long delays.

• The paper introduces a novel DHRL framework that leverages MDP homomorphisms to abstract augmented states for effective RL under delayed feedback.
• It compresses the exponential state augmentation by mapping belief-equivalent histories to a compact abstract MDP, ensuring optimal policy liftability.
• Empirical results on MuJoCo benchmarks via DHVI and D²HPG validate enhanced sample efficiency and robust performance even as delays increase.

Delayed Homomorphic Reinforcement Learning under Delayed Feedback

Problem Setting and Limitations of Canonical Approaches

Reinforcement learning (RL) in environments with delayed feedback presents a significant challenge due to the breakdown of the Markov property. When feedback is delayed, the environment's underlying dynamics become non-Markovian from the perspective of the agent, rendering standard RL algorithms sample-inefficient or, in many cases, unstable. The canonical state augmentation approach—where recent actions and observations are appended to the state to re-establish Markovian structure—suffers from exponential growth of the augmented state space with increasing delay length, directly escalating sample complexity and computational cost. This burden is particularly acute for actor-critic methods, where recent mitigation strategies primarily focus on the critic (e.g., belief projections) or rely on non-unified actor-critic treatments, leading to persistent inefficiencies for the policy optimization step.

DHRL: A Unified Homomorphic Abstraction Framework

This paper introduces Delayed Homomorphic Reinforcement Learning (DHRL), a principled framework leveraging Markov Decision Process (MDP) homomorphisms for sample-efficient RL in delayed feedback environments. DHRL replaces the blunt state augmentation with a theoretically grounded abstraction based on a belief-equivalence relation defined over the augmented state space. Two augmented states are identified as equivalent if they induce identical beliefs over the underlying physical state; this equivalence naturally collapses redundant distinctions among histories that are information-theoretically irrelevant.

The consequence is a homomorphic image of the original augmented MDP—a compact abstract MDP—on which both policy and value optimization can be performed. The framework guarantees that any optimal (or near-optimal) policy learned in the abstract MDP is liftable to an optimal policy for the original delayed MDP without loss of optimality. The theoretical results clarify that, under deterministic dynamics, the abstract state space is upper bounded by the original (delay-free) state space, totally eliminating the exponential dependence on delay (see Proposition 2, Corollary 2). In stochastic settings, further relaxation using total variation distance enables partial aggregation with bounded approximation error, effectively controlling sample complexity growth.

Theoretical Contributions: State and Sample Complexity Compression

The compactness of the belief-induced abstract MDP is key to the approach's efficiency. Under deterministic environment dynamics, the compression ratio (abstract to augmented state space size) is upper bounded as

$$\zeta := \frac{\lvert\bar{\mathcal{X}}\rvert}{\lvert\mathcal{X}\rvert} \leq \frac{1}{|\mathcal{A}|^{\Delta}}$$

where $\Delta$ is the feedback delay and $\mathcal{A}$ the action set. This result underpins a delay-independent sample complexity for $Q$-learning on the abstract MDP, with the complexity bound matching that of the original delay-free setting.

Figure 1: Normalized number of Bellman backups required for convergence using value iteration in regular (naive) and abstract (DHRL) MDPs with varying delays.

Empirical validation in a controlled grid world confirms theoretical scaling: the regular augmented MDP state-action space grows exponentially with $\Delta$, while the abstract space remains constant (see Figure 1), preserving the merits of canonical RL but at a dramatically reduced computational load.

Instantiations: DHVI and D$^2$HPG Algorithms

For finite, discrete MDPs, DHRL is instantiated as Delayed Homomorphic Value Iteration (DHVI). DHVI explicitly constructs the abstract MDP via belief equivalence and performs value iteration with fast convergence, validating the theoretical compression bounds.

For high-dimensional continuous control, the framework is realized as Deep Delayed Homomorphic Policy Gradient (D$^2$HPG), leveraging deep neural architectures to parameterize the homomorphism and actor-critic components. D$^2$HPG relies on the stochastic homomorphic policy gradient theorem, which ensures that policy gradients computed in the abstract MDP are valid surrogates for those in the original delayed MDP. Practical learning is achieved via amortized neural mapping between augmented and abstract states and actions, using auxiliary alignment losses to maintain consistency between the learned policy in the abstract MDP and the operational policy in the environment.

Figure 3: Schematic overview of D$^2$HPG, illustrating the homomorphic mapping from regular (augmented) to abstract (delay-free) state/action spaces.

Figure 2: Visual illustration of MuJoCo control tasks used for empirical validation, emphasizing algorithm robustness across diverse dynamical regimes.

Empirical Results: MuJoCo Continuous Control Benchmarks

Comprehensive evaluations on standard MuJoCo continuous control tasks (e.g., Ant-v3, HalfCheetah-v3, Hopper-v3, Humanoid-v3, etc.) demonstrate robust and superior performance of D$^2$HPG over state-of-the-art delayed-RL baselines, including naive and augmented SAC, BPQL, VDPO, and model-based methods. The performance advantage strengthens as delay increases, supporting the theoretical claims regarding sample efficiency and state-space compactness.

Figure 4: Normalized performance (return) of augmented SAC and D$\Delta$0HPG-naive as delay increases in HalfCheetah-v3. D$\Delta$1HPG shows less degradation and faster convergence.

Figure 5: Average return curves for each method as function of training time on MuJoCo benchmarks with delay $\Delta$2.

Figure 6: Average return curves for each method on MuJoCo benchmarks with increased delay ($\Delta$3).

Figure 7: Performance under long-delay regime ($\Delta$4), D$\Delta$5HPG remains competitive and stable where baselines degrade sharply.

Crucially, D$\Delta$6HPG can serve as a plug-in wrapper for actor-critic learning, offering further improvements when composed with sample-efficient baselines (e.g., BPQL). An ablation study shows that architectures leveraging both policy-alignment and abstract-gradient pathways yield the best trade-off between sample and computational efficiency.

Figure 9: Wall-clock time comparison. D$\Delta$7HPG is substantially faster than delayed model-based methods and scales favorably in large state spaces.

Theoretical and Practical Implications

The use of MDP homomorphisms for delay compensation fundamentally shifts the architecture design for RL with delayed (and even partially observable) environments. DHRL achieves strong sample efficiency, eliminates or severely reduces the curse of dimensionality typical of state augmentation, and presents a unified approach for both actor and critic.

Practically, the approach enables RL to scale to domains with substantial and variable delay, a longstanding limitation in robotics, networked control, and other cyberphysical applications. The main theoretical limitation is the assumption of fixed delays; however, the framework can be extended to random-bounded delays by employing surrogate constant-delay MDPs with modest empirical penalty.

Conclusion

The DHRL framework introduces a unification of belief-abstraction and MDP homomorphism theory for RL under feedback delays, closing key gaps left by augmentation-based methods. The main methodological innovation is the principled, provably minimal abstraction resulting in unified, sample-efficient training for both actor and critic, with strong numerical results across continuous control. Future work can extend the abstraction principle to stochastic dynamics and broader classes of non-Markovian structure, further expanding the reach of RL to real-world temporally challenging domains.

Figure 8: Robustness of D$\Delta$8HPG to random delays, supporting practical deployment beyond fixed-delay assumptions.