HR3L: Robust Remote Reinforcement Learning

• HR3L is a remote RL framework that employs a transmitter-receiver separation to communicate control-relevant, homomorphic latent state abstractions over impaired wireless channels.
• It uses a push-based protocol where the transmitter encodes unpredictable state features and the receiver predicts missing or delayed latents to maintain robust performance.
• Empirical results demonstrate HR3L’s superiority over standard PPO, with improved cumulative rewards under conditions of packet loss, delay, and capacity constraints.

Searching arXiv for papers on HR3L and closely related encrypted/robust remote RL work. Homomorphic Robust Remote Reinforcement Learning (HR3L) denotes a remote reinforcement-learning architecture for communication-impaired settings in which a transmitter colocated with the environment learns and communicates task-relevant latent state representations to a receiver colocated with the controller or actuator, which then selects actions to maximize return over a non-ideal wireless channel (Talli et al., 11 Aug 2025). In the defining formulation, HR3L addresses packet erasures, delay, intermittent observations, and limited capacity without requiring gradient exchange across the channel, and its “homomorphic” component refers to an MDP-homomorphism-inspired latent abstraction rather than homomorphic encryption (Talli et al., 11 Aug 2025). The term also sits at the intersection of earlier literatures on encrypted cloud RL updates, encrypted observation channels, and certified robust RL, but those works address only partial aspects of what HR3L names as a unified remote-learning problem (Suh et al., 2020, Suh et al., 2021, Jesu et al., 2021, Yang et al., 2023, Suh et al., 14 Jun 2025, Suh et al., 12 Apr 2025).

1. Definition, scope, and terminological boundaries

HR3L is introduced as a push-based remote RL framework for training agents over communication networks in which the acting unit does not have direct, timely access to the true state because observations traverse a lossy or delayed wireless link (Talli et al., 11 Aug 2025). The architecture has two physically separated units: a transmitter that directly observes the true environment state $s_t$, and a receiver that must choose actions $a_t$ to maximize reward after receiving possibly delayed or missing messages. The paper explicitly positions HR3L against prior approaches that either rely on explicit world models with substantial computational burden, treat impaired observations without communication-aware encoding, use pull-based formulations instead of stronger push-based ones, assume ideal channels or no packet loss or delay, or require differentiable channel models or gradient exchange (Talli et al., 11 Aug 2025).

A central terminological point is that “homomorphic” in HR3L does not mean ciphertext-domain computation. The method is motivated by exact MDP homomorphism conditions,

$r'(\sigma_s(s),\sigma_a(a)) = r(s,a),$

$$P'(\sigma_s(s')\mid \sigma_s(s),\sigma_a(a)) = P(s'\mid s,a),$$

but operationally it learns latent state and action embeddings that approximately preserve reward and dynamics relevant to control (Talli et al., 11 Aug 2025). The paper states that HR3L does not implement an exact symbolic homomorphism construction and is not presented as a bisimulation metric method; instead, it uses learned latent abstractions inspired by MDP homomorphisms and successor feature representations (Talli et al., 11 Aug 2025).

This terminological boundary matters because adjacent encrypted-RL papers use “homomorphic” in the cryptographic sense. In “SARSA(0) Reinforcement Learning over Fully Homomorphic Encryption” (Suh et al., 2020) and “Encrypted Value Iteration and Temporal Difference Learning over Leveled Homomorphic Encryption” (Suh et al., 2021), the cloud computes RL updates on CKKS ciphertexts. In HR3L proper, by contrast, the main challenge is not ciphertext arithmetic but remote state communication under packet loss, delay, and bandwidth limits (Talli et al., 11 Aug 2025).

2. Remote MDP formulation and distributed control architecture

The formal environment is an infinite-horizon Remote MDP,

$$\langle \mathcal{S}, \mathcal{A}, \mathcal{M}, \mathcal{C}, P, r, \gamma \rangle,$$

where $\mathcal{S}$ is the state space, $\mathcal{A}$ the action space, $\mathcal{M}$ the message space, $\mathcal{C}$ the communication channel model, $P$ the transition kernel, $a_t$0 the reward function, and $a_t$1 the discount factor (Talli et al., 11 Aug 2025). At time $a_t$2, the transmitter observes $a_t$3, sends message $a_t$4, the receiver obtains a delayed or lost version $a_t$5, selects $a_t$6, and the environment returns $a_t$7. The receiver seeks a control policy

$a_t$8

while the transmitter learns an encoding policy

$a_t$9

with $r'(\sigma_s(s),\sigma_a(a)) = r(s,a),$0 for $r'(\sigma_s(s),\sigma_a(a)) = r(s,a),$1-bit messages (Talli et al., 11 Aug 2025).

The joint communication-control problem is written as

$r'(\sigma_s(s),\sigma_a(a)) = r(s,a),$2

where $r'(\sigma_s(s),\sigma_a(a)) = r(s,a),$3 is channel capacity and the conditional mutual information

$r'(\sigma_s(s),\sigma_a(a)) = r(s,a),$4

measures the new information about $r'(\sigma_s(s),\sigma_a(a)) = r(s,a),$5 provided by $r'(\sigma_s(s),\sigma_a(a)) = r(s,a),$6 beyond what the receiver can already infer from prior messages (Talli et al., 11 Aug 2025). The formulation reflects the paper’s claim that, because states are temporally correlated, only the unpredictable part of the state should be communicated.

The channel model introduces three impairments: capacity constraints, delay, and packet erasures. For packet loss, the channel is modeled as a Packet Erasure Channel with state-dependent erasure probability following a Gilbert–Elliott two-state Markov chain; the simulations use

$r'(\sigma_s(s),\sigma_a(a)) = r(s,a),$7

with bad-state loss probabilities $r'(\sigma_s(s),\sigma_a(a)) = r(s,a),$8 and $r'(\sigma_s(s),\sigma_a(a)) = r(s,a),$9, corresponding to average success rates $$P'(\sigma_s(s')\mid \sigma_s(s),\sigma_a(a)) = P(s'\mid s,a),$$0 and $$P'(\sigma_s(s')\mid \sigma_s(s),\sigma_a(a)) = P(s'\mid s,a),$$1 (Talli et al., 11 Aug 2025). For delay, experiments use deterministic constant delays $$P'(\sigma_s(s')\mid \sigma_s(s),\sigma_a(a)) = P(s'\mid s,a),$$2, corresponding to $$P'(\sigma_s(s')\mid \sigma_s(s),\sigma_a(a)) = P(s'\mid s,a),$$3, $$P'(\sigma_s(s')\mid \sigma_s(s),\sigma_a(a)) = P(s'\mid s,a),$$4, and $$P'(\sigma_s(s')\mid \sigma_s(s),\sigma_a(a)) = P(s'\mid s,a),$$5 ms at $$P'(\sigma_s(s')\mid \sigma_s(s),\sigma_a(a)) = P(s'\mid s,a),$$6 ms (Talli et al., 11 Aug 2025).

The information split is asymmetric. The transmitter directly observes $$P'(\sigma_s(s')\mid \sigma_s(s),\sigma_a(a)) = P(s'\mid s,a),$$7 but does not directly observe $$P'(\sigma_s(s')\mid \sigma_s(s),\sigma_a(a)) = P(s'\mid s,a),$$8 at every step unless fed back; the receiver observes rewards, maintains a latent estimate of the current state, and receives periodic updates of the learned latent transition model $$P'(\sigma_s(s')\mid \sigma_s(s),\sigma_a(a)) = P(s'\mid s,a),$$9 and action embedding $$\langle \mathcal{S}, \mathcal{A}, \mathcal{M}, \mathcal{C}, P, r, \gamma \rangle,$$0 from the transmitter (Talli et al., 11 Aug 2025). This two-timescale architecture is a defining feature of HR3L.

3. Homomorphic state representations and latent control structure

HR3L learns a state embedding

$$\langle \mathcal{S}, \mathcal{A}, \mathcal{M}, \mathcal{C}, P, r, \gamma \rangle,$$1

an action embedding

$$\langle \mathcal{S}, \mathcal{A}, \mathcal{M}, \mathcal{C}, P, r, \gamma \rangle,$$2

and a combined latent state-action vector

$$\langle \mathcal{S}, \mathcal{A}, \mathcal{M}, \mathcal{C}, P, r, \gamma \rangle,$$3

together with a latent transition matrix

$$\langle \mathcal{S}, \mathcal{A}, \mathcal{M}, \mathcal{C}, P, r, \gamma \rangle,$$4

and a reward weight vector

$$\langle \mathcal{S}, \mathcal{A}, \mathcal{M}, \mathcal{C}, P, r, \gamma \rangle,$$5

such that

$$\langle \mathcal{S}, \mathcal{A}, \mathcal{M}, \mathcal{C}, P, r, \gamma \rangle,$$6

approximately hold (Talli et al., 11 Aug 2025). The representation is therefore trained to preserve reward prediction and latent next-state prediction rather than raw observation reconstruction.

The paper explicitly connects this design to successor feature representations. It recalls

$$\langle \mathcal{S}, \mathcal{A}, \mathcal{M}, \mathcal{C}, P, r, \gamma \rangle,$$7

represents reward as

$$\langle \mathcal{S}, \mathcal{A}, \mathcal{M}, \mathcal{C}, P, r, \gamma \rangle,$$8

defines the successor feature representation

$$\langle \mathcal{S}, \mathcal{A}, \mathcal{M}, \mathcal{C}, P, r, \gamma \rangle,$$9

and writes

$\mathcal{S}$0

The notation is compressed in the source, but the stated purpose is to build a control-relevant latent space in which reward and transition prediction are linear (Talli et al., 11 Aug 2025).

This structural choice underlies the receiver’s robustness mechanism. Because the latent state and reward obey approximately linear dynamics, the receiver can predict missing or delayed latent states using the current latent estimate and the known action sequence, rather than relying on raw observations (Talli et al., 11 Aug 2025). A common misconception is to equate HR3L’s “homomorphic” label with encrypted computation. The defining paper instead treats homomorphism as an abstraction principle: the transmitter learns the smallest predictive control-relevant latent summary of the environment, and the receiver acts in that latent space (Talli et al., 11 Aug 2025).

4. Asynchronous training protocol and communication workflow

Training proceeds in alternating rounds, with the transmitter and receiver updated locally and only compact summaries exchanged between them (Talli et al., 11 Aug 2025). The receiver uses PPO over latent state estimates, with rollout buffer

$\mathcal{S}$1

while the transmitter accumulates

$\mathcal{S}$2

and updates $\mathcal{S}$3 by minimizing

$\mathcal{S}$4

with target-network update

$\mathcal{S}$5

(Talli et al., 11 Aug 2025).

The per-step loop is: the transmitter observes $\mathcal{S}$6, computes $\mathcal{S}$7, forms message $\mathcal{S}$8, and sends either the full latent state or a masked version; the receiver decodes or predicts $\mathcal{S}$9, samples $\mathcal{A}$0, receives $\mathcal{A}$1, and stores the transition (Talli et al., 11 Aug 2025). At round end, the receiver sends the action and reward history of the round back to the transmitter; the transmitter updates $\mathcal{A}$2 and sends the updated $\mathcal{A}$3 to the receiver; the receiver then updates PPO using the current round’s latent rollouts (Talli et al., 11 Aug 2025).

The paper repeatedly stresses that HR3L does not exchange gradients over the wireless channel and does not require a differentiable channel model (Talli et al., 11 Aug 2025). This is one of its principal systems claims. The receiver algorithm is specifically chosen as PPO because it is on-policy and can adapt more easily when the latent representation changes round by round; the rollout buffer must be reset when encoding changes (Talli et al., 11 Aug 2025).

Implementation details reported for the receiver are PPO learning rate $\mathcal{A}$4, $\mathcal{A}$5, batch size $\mathcal{A}$6, $\mathcal{A}$7, use_sde $\mathcal{A}$8 True, sde_sample_freq $\mathcal{A}$9, and network architecture $\mathcal{M}$0 (Talli et al., 11 Aug 2025). For the transmitter, the paper reports learning rate $\mathcal{M}$1, horizon $\mathcal{M}$2, batch size $\mathcal{M}$3, $\mathcal{M}$4 epochs, and buffer size $\mathcal{M}$5 (Talli et al., 11 Aug 2025). Vector-state encoders are FNNs, image-state encoders are CNNs, and for visual tasks the paper uses $\mathcal{M}$6 features with $\mathcal{M}$7 bits per feature (Talli et al., 11 Aug 2025).

5. Robustness mechanisms under delay, loss, and bandwidth limits

HR3L’s robustness target is communication robustness rather than adversarial robustness. When a packet is lost, the receiver predicts the current latent state via

$\mathcal{M}$8

When a delayed packet for time $\mathcal{M}$9 arrives, the receiver fuses transmitted and predicted latent components using a binary mask $\mathcal{C}$0: $\mathcal{C}$1 then rolls this posterior estimate forward using the latent transition model and the recorded actions (Talli et al., 11 Aug 2025). These two equations constitute the core delay- and erasure-robust mechanism.

Under capacity constraints, if each feature uses $\mathcal{C}$2 bits and the channel budget is $\mathcal{C}$3 bits, only

$\mathcal{C}$4

features can be transmitted (Talli et al., 11 Aug 2025). HR3L therefore uses a binary mask

$\mathcal{C}$5

and selects the $\mathcal{C}$6 coordinates with largest receiver prediction error, where

$\mathcal{C}$7

is computed elementwise (Talli et al., 11 Aug 2025). The paper interprets this as goal-oriented communication: the transmitter sends the latent dimensions the receiver is worst at predicting.

Empirically, the paper evaluates HR3L on 25 tasks from the DeepMind Control Suite and reports that, under ideal communication, HR3L improves normalized cumulative reward by $\mathcal{C}$8 over standard PPO (Talli et al., 11 Aug 2025). Under packet loss, mean relative performance loss versus ideal communication is reported as $\mathcal{C}$9 and $P$0 for PPO under GE 95.5 and GE 92.5, versus $P$1 and $P$2 for HR3L (Talli et al., 11 Aug 2025). Under delay, average performance degradation relative to zero delay is $P$3, $P$4, and $P$5 for PPO at $P$6, $P$7, and $P$8 ms, compared with $P$9, $a_t$00, and $a_t$01 for HR3L; the paper states that HR3L gives about a $a_t$02 gain over PPO for $a_t$03 steps (Talli et al., 11 Aug 2025).

For visual RL, the paper reports that raw image transmission requires approximately $a_t$04 Mb/s, whereas uncompressed HR3L latent transmission requires approximately $a_t$05 Mb/s (Talli et al., 11 Aug 2025). In visual cheetah-run, HR3L learns almost as fast as DrQ-v2 and reaches similar total reward, whereas PPO barely reaches $a_t$06 of maximum performance at training end (Talli et al., 11 Aug 2025). Mean compression/decompression delay is reported as $a_t$07 ms for HR3L, $a_t$08 ms for CompressAI, and $a_t$09 ms for JPEG (Talli et al., 11 Aug 2025). The paper therefore attributes HR3L’s empirical behavior to the combination of low-dimensional control-relevant latents, latent dynamics prediction, selective communication of hard-to-predict features, and elimination of gradient exchange across the wireless link.

6. Relation to homomorphic encryption, encrypted observations, and certified robustness

HR3L sits near several adjacent research lines, but these lines use different meanings of privacy, robustness, and remote learning. In the cloud-RL-over-encryption literature, “SARSA(0) Reinforcement Learning over Fully Homomorphic Encryption” (Suh et al., 2020) studies a cloud-based RL architecture in which the client keeps plant interaction and action execution local, outsources tabular SARSA value updates to the cloud under CKKS encryption, and introduces delayed SARSA(0) with blocking to handle fixed computation latency. “Encrypted Value Iteration and Temporal Difference Learning over Leveled Homomorphic Encryption” (Suh et al., 2021) similarly analyzes remote tabular RL updates under CKKS, proves that encrypted value iteration converges to a neighborhood bounded by $a_t$10, and demonstrates TD(0), SARSA(0), and Z-learning over leveled homomorphic encryption. These works are foundational for secure outsourced RL updates, but their robustness notion is robustness to encryption-induced numerical noise or delay, not the wireless-communication robustness that defines HR3L (Suh et al., 2020, Suh et al., 2021).

A second nearby line seeks FHE-friendly RL formulations by removing comparison operations. “Relative Entropy Regularized Reinforcement Learning for Efficient Encrypted Policy Synthesis” (Suh et al., 14 Jun 2025) and “Efficient Implementation of Reinforcement Learning over Homomorphic Encryption” (Suh et al., 12 Apr 2025) both argue that standard RL is structurally mismatched with FHE because $a_t$11, $a_t$12, $a_t$13, and $a_t$14 are difficult to evaluate over encrypted data. They therefore focus on relative-entropy-regularized RL, linearly solvable value iteration, path integral control, and Z-learning, where the core updates are comparison-free or linear after a desirability transformation (Suh et al., 14 Jun 2025, Suh et al., 12 Apr 2025). This suggests a different route toward “homomorphic remote RL”: not latent-state abstraction over a wireless link, but privacy-preserving offline policy synthesis over ciphertexts.

Other adjacent works isolate still different subproblems. “Reinforcement Learning on Encrypted Data” (Jesu et al., 2021) modifies DQN so that the agent receives encrypted observations $a_t$15 instead of plaintext states and shows that learning is possible in small discrete environments but fails in more complex ones, while CKKS encryption is two to three orders of magnitude slower than simpler transformations (Jesu et al., 2021). “Certifiably Robust Reinforcement Learning through Model-Based Abstract Interpretation” (Yang et al., 2023) addresses adversarially perturbed observations and provides machine-checkable certificates of worst-case cumulative reward lower bounds, but it does not address homomorphic encryption, remote communication constraints, or client-server secure execution (Yang et al., 2023). “HERL: Tiered Federated Learning with Adaptive Homomorphic Encryption using Reinforcement Learning” (Tang et al., 2024) uses Q-learning to select CKKS parameter plans $a_t$16 in heterogeneous federated learning, which is directly relevant only if RL is interpreted as a controller for a secure remote learning system rather than the protected learning task itself (Tang et al., 2024).

Taken together, these papers clarify a persistent ambiguity around HR3L. One meaning, instantiated by (Talli et al., 11 Aug 2025), is a wireless remote-RL system based on homomorphic state representations and communication robustness. Another meaning, developed across (Suh et al., 2020, Suh et al., 2021, Suh et al., 14 Jun 2025), and (Suh et al., 12 Apr 2025), concerns remote RL or policy synthesis over homomorphic encryption. A plausible implication is that future work may combine these strands, but the current literature treats them largely as separate design problems.

7. Limitations, misconceptions, and open directions

The defining HR3L paper is methodological and empirical rather than theorem-heavy: it does not present a new formal convergence theorem or an end-to-end optimality guarantee for the full training algorithm (Talli et al., 11 Aug 2025). The paper also notes or implies several limitations: no stochastic delay modeling yet, a feedback channel assumed effectively ideal or of small overhead, no formal guarantee for approximate homomorphic latent sufficiency, and round-wise representation drift requiring rollout-buffer resets (Talli et al., 11 Aug 2025). Additional implied failure modes are that prediction under loss or delay may fail if the learned latent model $a_t$17 is poor, approximation quality may suffer if task-relevant information is not well captured by linear latent reward and dynamics structure, and peak rate-distortion performance in visual compression can be slightly inferior to specialized learned codecs such as CompressAI (Talli et al., 11 Aug 2025).

A common misconception is to read HR3L as a solved synthesis of homomorphic encryption, robust RL, and remote learning in a single protocol. The literature does not support that interpretation. The wireless HR3L framework does not use ciphertext-domain homomorphic encryption (Talli et al., 11 Aug 2025); the encrypted cloud-RL papers are mostly tabular, typically keep action selection local after decryption, and do not address packet loss, bursty wireless delay, or end-to-end remote actor-learner operation (Suh et al., 2020, Suh et al., 2021). Likewise, the certified-robustness literature defines robustness as worst-case performance under adversarial perturbations rather than resilience to delayed or intermittent observations (Yang et al., 2023).

The open research agenda stated or implied across these papers includes robustness with impaired reward or action feedback channels, stochastic delays and more realistic wireless models, stronger theory for approximate homomorphic latent sufficiency, extension to other RL backends and multi-agent settings, and better feature-selection rules than squared prediction error heuristics in the wireless HR3L formulation (Talli et al., 11 Aug 2025). In the encryption-oriented strand, open problems include encrypted policy improvement without difficult comparisons, richer function approximators, variable latency, stronger threat models than semi-honest clouds, and formal analyses of approximation error propagation in model-free learning (Suh et al., 2020, Suh et al., 2021, Suh et al., 12 Apr 2025, Suh et al., 14 Jun 2025). This suggests that HR3L is presently best understood not as a finished paradigm but as a family resemblance across remote RL, task-preserving abstraction, communication robustness, and privacy-preserving outsourced learning.