Rennala NIGT: Asynchronous Distributed RL

• Rennala NIGT is an asynchronous distributed policy-gradient method designed for homogeneous settings where agents share the same environment, reward function, and policy class.
• It integrates NIGT-style momentum and extrapolation with a single-update communication scheme to address straggler effects and reduce communication overhead.
• Empirical results on MuJoCo tasks show that Rennala NIGT converges faster and is more robust under heterogeneous computation speeds and significant communication delays.

Searching arXiv for the specified paper and closely related work on Rennala/Malenia methods. arXiv search query: "Asynchronous Policy Gradient Aggregation for Efficient Distributed Reinforcement Learning" OR "Rennala NIGT" Rennala NIGT is an asynchronous distributed policy-gradient method for homogeneous reinforcement learning, introduced for settings in which all agents interact with the same environment, reward function, and policy class, yet may have different computation speeds and incur nontrivial communication delays. Its defining mechanism is asynchronous aggregation of many local stochastic policy gradients, followed by a single communication event per global update, combined with a NIGT-style extrapolation-and-momentum scheme and a normalized parameter update. In the theory developed for discounted RL, it improves prior asynchronous distributed policy-gradient guarantees in both computational and communication complexity while remaining compatible with AllReduce (Tyurin et al., 29 Sep 2025).

1. Problem setting and meaning of “homogeneous”

The method is formulated for discounted reinforcement learning with objective

$$\max_{\theta\in\mathbb{R}^d} J(\theta) = \mathbb{E}\!\left[\sum_{t=0}^{\infty}\gamma^t r(s_t,a_t)\right].$$

Each agent samples finite trajectories $\tau=(s_0,a_0,\dots,s_{H-1},a_{H-1})$ and works with the truncated objective

$$J_H(\theta)=\mathbb{E}_{\tau}\left[\sum_{t=0}^{H-1}\gamma^t r(s_t,a_t)\right].$$

For one trajectory, the stochastic policy-gradient estimator is

$$g_H(\tau,\theta)=\sum_{t=0}^{H-1}\left(\sum_{h=t}^{H-1}\gamma^h r(s_h,a_h)\right)\nabla\log \pi_\theta(a_t\mid s_t),$$

and it is unbiased for the truncated gradient: $$\nabla J_H(\theta)=\mathbb{E}_\tau[g_H(\tau,\theta)].$$

In this setting, “homogeneous” does not mean equal-speed workers or negligible systems effects. It means that all agents share the same environment, reward function, and policy class. The distributed difficulty arises elsewhere: workers finish at different times, strict synchronization forces the system to wait for the slowest worker, and fully asynchronous schemes that transmit every gradient individually can make communication the bottleneck and may not support efficient collective communication such as AllReduce. Rennala NIGT addresses that systems-level bottleneck by aggregating many local stochastic gradients asynchronously and communicating only once per global update (Tyurin et al., 29 Sep 2025).

2. Algorithmic structure

Rennala NIGT combines a NIGT-style extrapolation/momentum mechanism with a custom asynchronous aggregation subroutine. The global procedure starts from $\theta_0$, chooses momentum $\eta$, step size $\alpha$, horizon $H$, and batch sizes $M_{\text{init}}$ and $\tau=(s_0,a_0,\dots,s_{H-1},a_{H-1})$0. It computes an initial aggregated direction and then takes the first normalized step

$\tau=(s_0,a_0,\dots,s_{H-1},a_{H-1})$1

For each iteration $\tau=(s_0,a_0,\dots,s_{H-1},a_{H-1})$2, the method forms the extrapolated point

$\tau=(s_0,a_0,\dots,s_{H-1},a_{H-1})$3

which is the NIGT-style “lookahead” point. It then computes

$\tau=(s_0,a_0,\dots,s_{H-1},a_{H-1})$4

updates the momentum variable by

$\tau=(s_0,a_0,\dots,s_{H-1},a_{H-1})$5

and performs the normalized parameter update

$\tau=(s_0,a_0,\dots,s_{H-1},a_{H-1})$6

with the convention $\tau=(s_0,a_0,\dots,s_{H-1},a_{H-1})$7.

The asynchronous aggregation subroutine is the operational core. The coordinator broadcasts $\tau=(s_0,a_0,\dots,s_{H-1},a_{H-1})$8 to all agents; every agent starts sampling trajectories and computing local stochastic gradients $\tau=(s_0,a_0,\dots,s_{H-1},a_{H-1})$9; the coordinator waits for gradients to arrive from whichever agent finishes first; each arriving gradient is added to a running sum, and that agent immediately starts the next trajectory; after $$J_H(\theta)=\mathbb{E}_{\tau}\left[\sum_{t=0}^{H-1}\gamma^t r(s_t,a_t)\right].$$0 stochastic gradients have been collected in total, the algorithm stops all workers and returns the average

$$J_H(\theta)=\mathbb{E}_{\tau}\left[\sum_{t=0}^{H-1}\gamma^t r(s_t,a_t)\right].$$1

where $$J_H(\theta)=\mathbb{E}_{\tau}\left[\sum_{t=0}^{H-1}\gamma^t r(s_t,a_t)\right].$$2 are the $$J_H(\theta)=\mathbb{E}_{\tau}\left[\sum_{t=0}^{H-1}\gamma^t r(s_t,a_t)\right].$$3 trajectories collected in arrival order. A key implementation point is that this can be realized with a single AllReduce call at the end, because local partial sums can be aggregated collectively rather than through repeated server round-trips (Tyurin et al., 29 Sep 2025).

3. Regularity assumptions and estimator properties

The homogeneous theory assumes that the policy $$J_H(\theta)=\mathbb{E}_{\tau}\left[\sum_{t=0}^{H-1}\gamma^t r(s_t,a_t)\right].$$4 is positive and twice continuously differentiable, that the log-policy satisfies the derivative bounds

$$J_H(\theta)=\mathbb{E}_{\tau}\left[\sum_{t=0}^{H-1}\gamma^t r(s_t,a_t)\right].$$5

and that its Hessian is Lipschitz: $$J_H(\theta)=\mathbb{E}_{\tau}\left[\sum_{t=0}^{H-1}\gamma^t r(s_t,a_t)\right].$$6

These assumptions imply smoothness of the RL objective,

$$J_H(\theta)=\mathbb{E}_{\tau}\left[\sum_{t=0}^{H-1}\gamma^t r(s_t,a_t)\right].$$7

and also a second-order smoothness bound

$$J_H(\theta)=\mathbb{E}_{\tau}\left[\sum_{t=0}^{H-1}\gamma^t r(s_t,a_t)\right].$$8

The paper further states truncation and variance controls: $$J_H(\theta)=\mathbb{E}_{\tau}\left[\sum_{t=0}^{H-1}\gamma^t r(s_t,a_t)\right].$$9 and

$$g_H(\tau,\theta)=\sum_{t=0}^{H-1}\left(\sum_{h=t}^{H-1}\gamma^h r(s_h,a_h)\right)\nabla\log \pi_\theta(a_t\mid s_t),$$0

The aggregation mechanism preserves the essential statistical structure because, in the homogeneous setting, all collected trajectories are i.i.d. from the same distribution. Hence the aggregated estimator is unbiased for the truncated gradient,

$$g_H(\tau,\theta)=\sum_{t=0}^{H-1}\left(\sum_{h=t}^{H-1}\gamma^h r(s_h,a_h)\right)\nabla\log \pi_\theta(a_t\mid s_t),$$1

and its variance contracts with the batch size: $$g_H(\tau,\theta)=\sum_{t=0}^{H-1}\left(\sum_{h=t}^{H-1}\gamma^h r(s_h,a_h)\right)\nabla\log \pi_\theta(a_t\mid s_t),$$2 This is the central analytic reason the method can reduce communication frequency without sacrificing gradient-estimation quality: variance reduction comes from aggregating $$g_H(\tau,\theta)=\sum_{t=0}^{H-1}\left(\sum_{h=t}^{H-1}\gamma^h r(s_h,a_h)\right)\nabla\log \pi_\theta(a_t\mid s_t),$$3 asynchronously obtained samples, rather than from lock-step synchronization at the level of individual gradients (Tyurin et al., 29 Sep 2025).

4. Time model and convergence guarantees

The paper models the time to compute one stochastic gradient on agent $$g_H(\tau,\theta)=\sum_{t=0}^{H-1}\left(\sum_{h=t}^{H-1}\gamma^h r(s_h,a_h)\right)\nabla\log \pi_\theta(a_t\mid s_t),$$4 as

$$g_H(\tau,\theta)=\sum_{t=0}^{H-1}\left(\sum_{h=t}^{H-1}\gamma^h r(s_h,a_h)\right)\nabla\log \pi_\theta(a_t\mid s_t),$$5

where $$g_H(\tau,\theta)=\sum_{t=0}^{H-1}\left(\sum_{h=t}^{H-1}\gamma^h r(s_h,a_h)\right)\nabla\log \pi_\theta(a_t\mid s_t),$$6 is the per-environment-step time and $$g_H(\tau,\theta)=\sum_{t=0}^{H-1}\left(\sum_{h=t}^{H-1}\gamma^h r(s_h,a_h)\right)\nabla\log \pi_\theta(a_t\mid s_t),$$7 is the trajectory length. Communication time for sending a vector is denoted by $$g_H(\tau,\theta)=\sum_{t=0}^{H-1}\left(\sum_{h=t}^{H-1}\gamma^h r(s_h,a_h)\right)\nabla\log \pi_\theta(a_t\mid s_t),$$8, and the same parameter is used for centralized communication and decentralized collective operations such as AllReduce.

Under this model, the time to collect $$g_H(\tau,\theta)=\sum_{t=0}^{H-1}\left(\sum_{h=t}^{H-1}\gamma^h r(s_h,a_h)\right)\nabla\log \pi_\theta(a_t\mid s_t),$$9 gradients with Rennala’s asynchronous aggregator is bounded by

$$\nabla J_H(\theta)=\mathbb{E}_\tau[g_H(\tau,\theta)].$$0

The dependence is harmonic-mean-like: it allows the analysis to effectively ignore sufficiently slow workers by selecting the best subset size $$\nabla J_H(\theta)=\mathbb{E}_\tau[g_H(\tau,\theta)].$$1, which is the source of the method’s straggler robustness.

The main convergence theorem gives parameter choices for $$\nabla J_H(\theta)=\mathbb{E}_\tau[g_H(\tau,\theta)].$$2, $$\nabla J_H(\theta)=\mathbb{E}_\tau[g_H(\tau,\theta)].$$3, $$\nabla J_H(\theta)=\mathbb{E}_\tau[g_H(\tau,\theta)].$$4, $$\nabla J_H(\theta)=\mathbb{E}_\tau[g_H(\tau,\theta)].$$5, and $$\nabla J_H(\theta)=\mathbb{E}_\tau[g_H(\tau,\theta)].$$6, and states that the method reaches an $$\nabla J_H(\theta)=\mathbb{E}_\tau[g_H(\tau,\theta)].$$7-stationary point after a number of global iterations with terms

$$\nabla J_H(\theta)=\mathbb{E}_\tau[g_H(\tau,\theta)].$$8

where $$\nabla J_H(\theta)=\mathbb{E}_\tau[g_H(\tau,\theta)].$$9. The stated homogeneous total time bound is a $\theta_0$0 expression with a communication component proportional to

$\theta_0$1

and a computation component proportional to

$\theta_0$2

with the inner term including the variance-sensitive contributions $\theta_0$3 and $\theta_0$4. This separation makes explicit that one source of improvement comes from reducing the number of communication rounds, while another comes from a speed-aware aggregation rule that does not force every worker to determine each update time (Tyurin et al., 29 Sep 2025).

5. Comparison with distributed policy-gradient baselines

The paper positions Rennala NIGT against three baselines: synchronized Vanilla PG, synchronized NIGT, and AFedPG. The central point of comparison is not merely iteration complexity, but the joint computational and communication burden in asynchronous heterogeneous systems.

Method	Characteristic complexity feature	Distinguishing limitation or advantage
Synchronized Vanilla PG	$\theta_0$5 plus similar communication cost	Bottlenecked by the slowest worker
Synchronized NIGT	Communication cost $\theta_0$6 in the table	Less communication-efficient than Rennala NIGT
AFedPG	Communication $\theta_0$7, worst case $\theta_0$8	Does not support AllReduce
Rennala NIGT	$\theta_0$9	Supports AllReduce and improves communication complexity

Against synchronized Vanilla PG, Rennala NIGT avoids explicit dependence on the slowest worker and replaces it with a subset-adaptive harmonic-mean expression. Against synchronized NIGT, the paper’s claim is primarily about communication efficiency: synchronized NIGT inherits better centralized optimization rates, but distributed RL still pays a communication cost $\eta$0 in the table, whereas Rennala NIGT reduces the communication dependence to $\eta$1 in the small-$\eta$2 regime. Against AFedPG, the claimed advantages are twofold: communication improves from $\eta$3 to $\eta$4, and the computation term can decrease as additional workers are added instead of worsening with $\eta$5. The paper also emphasizes that Rennala NIGT supports AllReduce, while AFedPG does not (Tyurin et al., 29 Sep 2025).

6. Empirical behavior

The empirical study is conducted on MuJoCo tasks, with particular emphasis on Humanoid-v4, and also includes Reacher-v4, Walker2d-v4, and Hopper-v4. The reported qualitative pattern is systematic across regimes. When compute speeds are equal and there is no communication delay, Rennala NIGT, AFedPG, and synchronized NIGT behave similarly. Under heterogeneous computation times, Rennala NIGT converges faster than the baselines. When heterogeneity is combined with large communication delays, Rennala NIGT is reported as the most robust and the performance gap widens.

The paper also scales experiments to $\eta$6 agents. In that regime, equal-time settings again produce similar behavior across methods, while heterogeneous or communication-expensive settings preserve Rennala NIGT’s advantage. The strongest empirical gains are highlighted on Humanoid-v4 and Reacher-v4, where the method is often faster and its margin grows with systems heterogeneity.

The same paper additionally studies Malenia NIGT in a genuinely heterogeneous-environment experiment, where one agent sees normal Humanoid-v4 and another sees inverted states. That experiment is not evidence about Rennala NIGT itself, but it clarifies the boundary between the two algorithms: Rennala NIGT is the homogeneous-environment method, whereas Malenia NIGT is the heterogeneous extension (Tyurin et al., 29 Sep 2025).

7. Relation to broader asynchronous optimization literature

The name “Rennala” already appears in earlier distributed stochastic optimization. Rennala SGD is the optimal parallel stochastic gradient method for the homogeneous setup under a universal computation model that allows arbitrary worker-speed behavior, including changing speeds, outages, and heterogeneity. Its mechanism is a batching-and-waiting SGD scheme: all workers compute stochastic gradients at the current iterate, the method waits until a prescribed batch size is collected, averages the gradients, updates the model, and stops ongoing computations before the next round. In that setting, matching upper and lower bounds establish optimality up to constants (Tyurin, 2024).

Subsequent work narrows the scope of that optimality. In the presence of random compute times, especially heavy-tailed or failure-prone delays, MindFlayer SGD argues that Rennala SGD’s wait-for-batch strategy can perform arbitrarily badly, and proposes capped waiting and retry as a more robust alternative. The critique is specific to random-delay regimes and does not contradict deterministic-delay optimality; rather, it shows that fixed-time optimality does not automatically transfer to stochastic runtime models (Maranjyan et al., 2024). In parallel, work on asynchronous finite-sum optimization treats Rennala SGD as an important baseline but emphasizes that its $\eta$7-bounded variance assumption and small-$\eta$8 dependence can be suboptimal for nonconvex finite-sum structure, motivating variance-reduced alternatives such as Freya PAGE (Tyurin et al., 2024).

This broader literature helps delimit Rennala NIGT precisely. Its guarantees concern homogeneous distributed RL with asynchronous computations and communication delays, not arbitrary heterogeneous environments and not random-delay models of the kind studied by MindFlayer. A common misconception is to equate “homogeneous” with full synchronization or equal worker speed. In the paper’s terminology, homogeneous refers only to shared environment, reward function, and policy class; heterogeneous compute speeds and nontrivial communication delays are explicit parts of the target setting, and the method’s asynchronous aggregation plus single-update communication pattern is designed around that systems model (Tyurin et al., 29 Sep 2025).