Malenia NIGT: Heterogeneous Distributed RL

• Malenia NIGT is an asynchronous, distributed policy-gradient method tailored for heterogeneous RL settings by aggregating local gradients.
• It reduces communication overhead by locally accumulating many gradients before a single unbiased collective aggregation step, effectively mitigating stragglers.
• Empirical results indicate that Malenia NIGT outperforms prior methods under heterogeneous computation and diverse environment conditions.

Malenia NIGT is an asynchronous, distributed policy-gradient method for the heterogeneous reinforcement-learning setting, meaning that different workers may interact with different environments, use different rewards and policies, and run at different computation speeds. It was introduced together with Rennala NIGT in "Asynchronous Policy Gradient Aggregation for Efficient Distributed Reinforcement Learning" as a communication-efficient alternative to synchronized policy-gradient schemes and to earlier asynchronous approaches that do not support heterogeneity or have suboptimal communication complexity (Tyurin et al., 29 Sep 2025). Its defining design choice is to let workers locally aggregate many gradients asynchronously and then perform a single collective aggregation step, thereby reducing communication overhead, mitigating stragglers, and preserving an unbiased estimator for the average heterogeneous objective.

1. Problem formulation and motivation

The method is formulated for distributed RL under two simultaneous systems constraints: asynchronous computation and communication bottlenecks. In asynchronous deployments, workers finish rollout generation and gradient computation at different times, so globally synchronized policy-gradient methods incur idle time while waiting for the slowest worker. In communication-constrained settings, especially federated or decentralized ones, sending every gradient separately is expensive and can obstruct efficient collective primitives such as AllReduce (Tyurin et al., 29 Sep 2025).

Malenia NIGT targets the case in which the distributed system is heterogeneous not only in speed but also in data-generating process. The workers may have different local objectives $J_i$, rewards $r_i$, policies $\pi_{i,\theta}$, and transition kernels $\mathcal{P}_i$. This makes the aggregation problem structurally different from the homogeneous case handled by Rennala NIGT. In the homogeneous case, raw averaging across asynchronously accumulated gradients is sufficient; in the heterogeneous case, unbiasedness requires a modified averaging rule that respects worker-specific local objectives (Tyurin et al., 29 Sep 2025).

The paper positions Malenia NIGT against prior asynchronous RL methods, especially AFedPG, arguing that AFedPG does not support heterogeneous environments and has suboptimal communication complexity. The contrast with synchronized policy-gradient and synchronized NIGT baselines is equally central: synchronization after every stochastic gradient is simpler, but it is less communication-efficient and more sensitive to stragglers (Tyurin et al., 29 Sep 2025).

2. Algorithmic structure

At the outer-loop level, Malenia NIGT is a momentum-and-normalization policy-gradient method augmented with asynchronous batch aggregation. The update presented in the paper is

$$\widetilde{\theta}_t = \theta_t + \frac{1 - \eta}{\eta} (\theta_t - \theta_{t - 1}),$$

$$g_t = \textnormal{AggregateMalenia}(\widetilde{\theta}_t, M, H),$$

$$d_t = (1 - \eta) d_{t - 1} + \eta g_t, \qquad \theta_{t + 1} = \theta_t + \alpha \frac{d_t}{d_t}.$$

The paper describes the final step as the same conceptual normalized step used by NIGT: the update direction is normalized so that the step length remains robust (Tyurin et al., 29 Sep 2025).

The aggregation routine is the distinctive component. The current parameter $\theta$ is broadcast, after which all agents begin sampling trajectories and computing gradients asynchronously. When a worker finishes one gradient estimate, it immediately starts the next one; the procedure continues until the algorithm has accumulated enough total gradients. This execution model is designed to exploit fast workers without blocking on slow ones (Tyurin et al., 29 Sep 2025).

For worker $i$, the heterogeneous trajectory gradient estimator is

$$g_{i,H}(\tau_i,\theta)=\sum\limits^{H-1}_{t=0}\left(\sum\limits^{H-1}_{h=t}\gamma^h r_i(s_{i,h},a_{i,h})\right) \nabla\log\pi_{i,\theta}(a_{i,t}|s_{i,t}),$$

with $r_i$0. The truncation horizon $r_i$1 controls rollout length and enters both the approximation error and the time model (Tyurin et al., 29 Sep 2025).

This construction makes Malenia NIGT a policy-gradient method specialized to heterogeneous distributed rollout generation rather than a direct transplantation of asynchronous SGD. The stochastic objects being aggregated are trajectory-based estimators adapted to local environments, not merely iid stochastic gradients from a single common objective.

3. Heterogeneity-aware aggregation and unbiasedness

The returned aggregate in Malenia NIGT is

$r_i$2

where $r_i$3 is the sum of gradients collected by worker $r_i$4, and $r_i$5 is the number of gradients worker $r_i$6 contributed. The paper identifies this as the key heterogeneity-aware design choice (Tyurin et al., 29 Sep 2025).

Its role is to preserve unbiasedness for the average truncated heterogeneous objective. Specifically,

$r_i$7

This property is the formal reason the method can accommodate distinct local environments. If worker outputs were pooled without the per-worker normalization by $r_i$8, the resulting estimator would generally overweight faster workers and cease to represent the average heterogeneous objective. The paper treats this unbiased aggregation result as the principal structural distinction between Malenia NIGT and homogeneous asynchronous aggregation (Tyurin et al., 29 Sep 2025).

A common misunderstanding is to view Malenia NIGT as merely Rennala NIGT with unequal worker speeds. That is incorrect. Rennala NIGT addresses the homogeneous case, where all workers share the same environment and objective and the aggregate is simply the average of all collected gradients. Malenia NIGT alters the aggregation rule because heterogeneity is in the objective itself, not only in runtime (Tyurin et al., 29 Sep 2025).

4. Assumptions, regularity, and complexity guarantees

The analysis assumes standard smoothness and boundedness conditions for policy gradients. Under the paper’s assumptions, the policy-gradient map enjoys standard regularity, with gradient smoothness constant

$r_i$9

and Hessian Lipschitz constant $\pi_{i,\theta}$0. The truncated objective $\pi_{i,\theta}$1 approximates the infinite-horizon objective $\pi_{i,\theta}$2 according to

$\pi_{i,\theta}$3

The gradient estimator satisfies the usual unbiasedness and variance control for the truncated objective, and the paper states that the same local property holds for each $\pi_{i,\theta}$4 in the heterogeneous setting (Tyurin et al., 29 Sep 2025).

A central modeling device is the computation-time model

$\pi_{i,\theta}$5

where $\pi_{i,\theta}$6 is the time for one transition step and $\pi_{i,\theta}$7 is the trajectory length. Communication takes at most $\pi_{i,\theta}$8 seconds per vector transfer, including AllReduce. The paper explicitly notes that these assumptions are used to derive time complexity, while convergence itself does not require them (Tyurin et al., 29 Sep 2025).

For the heterogeneous setting, Theorem $\pi_{i,\theta}$9 states that Malenia NIGT achieves

$\mathcal{P}_i$0

The paper identifies this as its main heterogeneous time bound. It emphasizes that the computational dependence is “mean-like” rather than harmonic-like, reflecting the stronger difficulty of heterogeneous environments, while the communication term remains the same as in the homogeneous case (Tyurin et al., 29 Sep 2025).

5. Relations to Rennala NIGT, AFedPG, and adjacent Malenia methods

The structural differences among the main methods discussed in the paper can be summarized compactly as follows.

Method	Setting	Distinguishing property
Malenia NIGT	Heterogeneous RL	Per-worker normalized aggregation preserves unbiasedness
Rennala NIGT	Homogeneous RL	Asynchronous aggregation with AllReduce support
AFedPG	Asynchronous RL	Does not support heterogeneity; suboptimal communication complexity

The paper’s comparison is not purely algorithmic but also systems-theoretic. Rennala NIGT and Malenia NIGT both support AllReduce, whereas AFedPG is described as relying on a greedy update strategy that prevents AllReduce and yields worse communication complexity. In the homogeneous regime, the paper states that Rennala NIGT attains communication complexity $\mathcal{P}_i$1 in the small-$\mathcal{P}_i$2 regime, improving on AFedPG’s $\mathcal{P}_i$3, and notes that in an extreme case AFedPG may require $\mathcal{P}_i$4 communications (Tyurin et al., 29 Sep 2025).

The paper also proves a lower bound for black-box first-order methods using only unbiased stochastic gradients of an $\mathcal{P}_i$5-twice smooth function. The stated consequence is that, although Malenia NIGT and Rennala NIGT improve prior methods, a gap remains between the algorithmic $\mathcal{P}_i$6 communication dependence and a lower-bound term involving $\mathcal{P}_i$7. This situates the method as an advance rather than a final characterization of optimal communication dependence in distributed policy-gradient RL (Tyurin et al., 29 Sep 2025).

The designation “Malenia” also appears in two separate arXiv works with unrelated domains. Malenia SGD is an optimal method for heterogeneous distributed stochastic optimization under a universal computation model (Tyurin, 2024). Malenia without “NIGT” is a vision-language pre-training framework for 3D zero-shot lesion segmentation (Jiang et al., 2024). These are distinct methods and problem settings. A plausible implication is that the repeated naming reflects a family naming convention rather than a shared technical framework.

6. Empirical evidence, interpretation, and limitations

The experimental section in the RL paper evaluates the asynchronous aggregation design on MuJoCo tasks including Humanoid-v4, Reacher-v4, Walker2d-v4, and Hopper-v4. The reported findings are: when computation and communication times are equal, all methods perform similarly; under heterogeneous computation times, Rennala NIGT converges faster than AFedPG and synchronized NIGT; with large communication delays, Rennala NIGT becomes even more advantageous; and with $\mathcal{P}_i$8, it remains the fastest method in multiple heterogeneous scenarios (Tyurin et al., 29 Sep 2025).

Direct heterogeneous-environment evidence for Malenia NIGT is provided in a two-agent Humanoid-v4 experiment in which one agent observes the normal environment and the other receives inverted states. In that setting, Malenia NIGT significantly outperforms AFedPG, matching the paper’s claim that Malenia NIGT supports heterogeneous RL whereas AFedPG does not (Tyurin et al., 29 Sep 2025).

Several interpretive points follow from the paper’s results. First, Malenia NIGT is not presented as a generic asynchronous optimizer with RL used only as an application domain; it is explicitly adapted to policy-gradient trajectories and heterogeneous local environments. Second, the method’s communication robustness derives from local asynchronous accumulation followed by one collective step, not from eliminating collectives altogether. Third, the time model is a tool for wall-clock analysis rather than a prerequisite for convergence.

The paper also leaves open questions. The lower-bound discussion shows that theoretical communication optimality is unresolved. The heterogeneous time bound is stronger than earlier asynchronous RL guarantees, but it does not close the gap to the lower bound. This suggests that further work may be needed on communication-efficient policy-gradient methods whose upper bounds more tightly match the black-box lower-bound structure (Tyurin et al., 29 Sep 2025).