---
title: 'Rennala NIGT: Asynchronous Distributed RL'
url: https://www.emergentmind.com/topics/rennala-nigt
type: topic
---

# Rennala NIGT: Asynchronous Distributed RL

Searching arXiv for the specified paper and closely related work on Rennala/Malenia methods.
arXiv search query: `"Asynchronous Policy Gradient Aggregation for Efficient Distributed Reinforcement Learning" OR "Rennala NIGT"`
Rennala NIGT is an asynchronous distributed policy-gradient method for homogeneous reinforcement learning, introduced for settings in which all agents interact with the same environment, reward function, and policy class, yet may have different computation speeds and incur nontrivial communication delays. Its defining mechanism is asynchronous aggregation of many local stochastic policy gradients, followed by a single communication event per global update, combined with a NIGT-style extrapolation-and-momentum scheme and a normalized parameter update. In the theory developed for discounted RL, it improves prior asynchronous distributed policy-gradient guarantees in both computational and communication complexity while remaining compatible with AllReduce [2509.24305].

## 1. Problem setting and meaning of “homogeneous”

The method is formulated for discounted reinforcement learning with objective
\[
\max_{\theta\in\mathbb{R}^d} J(\theta) = \mathbb{E}\!\left[\sum_{t=0}^{\infty}\gamma^t r(s_t,a_t)\right].
\]
Each agent samples finite trajectories \(\tau=(s_0,a_0,\dots,s_{H-1},a_{H-1})\) and works with the truncated objective
\[
J_H(\theta)=\mathbb{E}_{\tau}\left[\sum_{t=0}^{H-1}\gamma^t r(s_t,a_t)\right].
\]
For one trajectory, the stochastic policy-gradient estimator is
\[
g_H(\tau,\theta)=\sum_{t=0}^{H-1}\left(\sum_{h=t}^{H-1}\gamma^h r(s_h,a_h)\right)\nabla\log \pi_\theta(a_t\mid s_t),
\]
and it is unbiased for the truncated gradient:
\[
\nabla J_H(\theta)=\mathbb{E}_\tau[g_H(\tau,\theta)].
\]

In this setting, “homogeneous” does not mean equal-speed workers or negligible systems effects. It means that all agents share the same environment, reward function, and policy class. The distributed difficulty arises elsewhere: workers finish at different times, strict synchronization forces the system to wait for the slowest worker, and fully asynchronous schemes that transmit every gradient individually can make communication the bottleneck and may not support efficient collective communication such as AllReduce. Rennala NIGT addresses that systems-level bottleneck by aggregating many local stochastic gradients asynchronously and communicating only once per global update [2509.24305].

## 2. Algorithmic structure

Rennala NIGT combines a NIGT-style extrapolation/momentum mechanism with a custom asynchronous aggregation subroutine. The global procedure starts from \(\theta_0\), chooses momentum \(\eta\), step size \(\alpha\), horizon \(H\), and batch sizes \(M_{\text{init}}\) and \(M\). It computes an initial aggregated direction and then takes the first normalized step
\[
\theta_1=\theta_0+\alpha \frac{d_0}{\|d_0\|}.
\]

For each iteration \(t\ge 1\), the method forms the extrapolated point
\[
\widetilde{\theta}_t=\theta_t+\frac{1-\eta}{\eta}(\theta_t-\theta_{t-1}),
\]
which is the NIGT-style “lookahead” point. It then computes
\[
g_t = \textnormal{AggregateRennala}(\widetilde{\theta}_t,M,H),
\]
updates the momentum variable by
\[
d_t=(1-\eta)d_{t-1}+\eta g_t,
\]
and performs the normalized parameter update
\[
\theta_{t+1}=\theta_t+\alpha \frac{d_t}{\|d_t\|},
\]
with the convention \(0/0=0\).

The asynchronous aggregation subroutine is the operational core. The coordinator broadcasts \(\theta\) to all agents; every agent starts sampling trajectories and computing local stochastic gradients \(g_H(\tau,\theta)\); the coordinator waits for gradients to arrive from whichever agent finishes first; each arriving gradient is added to a running sum, and that agent immediately starts the next trajectory; after \(M\) stochastic gradients have been collected in total, the algorithm stops all workers and returns the average
\[
\bar g = \frac{1}{M}\sum_{j=1}^{M} g_H(\bar\tau_j,\theta),
\]
where \(\bar\tau_j\) are the \(M\) trajectories collected in arrival order. A key implementation point is that this can be realized with a single AllReduce call at the end, because local partial sums can be aggregated collectively rather than through repeated server round-trips [2509.24305].

## 3. Regularity assumptions and estimator properties

The homogeneous theory assumes that the policy \(\pi_\theta(a\mid s)\) is positive and twice continuously differentiable, that the log-policy satisfies the derivative bounds
\[
\|\nabla \log \pi_\theta(a\mid s)\|\le M_g,\qquad \|\nabla^2 \log \pi_\theta(a\mid s)\|\le M_h,
\]
and that its Hessian is Lipschitz:
\[
\|\nabla^2 \log \pi_\theta(a\mid s)-\nabla^2 \log \pi_{\bar\theta}(a\mid s)\|
\le l_2\|\theta-\bar\theta\|.
\]

These assumptions imply smoothness of the RL objective,
\[
\|\nabla J(\theta)-\nabla J(\theta')\|\le L_g\|\theta-\theta'\|,
\qquad
L_g:=\frac{r_{\max}(M_g^2+M_h)}{(1-\gamma)^2},
\]
and also a second-order smoothness bound
\[
\|\nabla^2 J(\theta)-\nabla^2 J(\theta')\|\le L_h\|\theta-\theta'\|.
\]
The paper further states truncation and variance controls:
\[
\|\nabla J_H(\theta)-\nabla J(\theta)\|\le D_g\gamma^H,\qquad
\|\nabla^2 J_H(\theta)-\nabla^2 J(\theta)\|\le D_h\gamma^H,
\]
and
\[
\mathbb{E}\|g_H(\tau,\theta)-\nabla J_H(\theta)\|^2\le \sigma^2,
\qquad
\sigma^2:=\frac{r_{\max}^2 M_g^2}{(1-\gamma)^3}.
\]

The aggregation mechanism preserves the essential statistical structure because, in the homogeneous setting, all collected trajectories are i.i.d. from the same distribution. Hence the aggregated estimator is unbiased for the truncated gradient,
\[
\mathbb{E}[\bar g]=\nabla J_H(\theta),
\]
and its variance contracts with the batch size:
\[
\mathbb{E}\|\bar g-\nabla J_H(\theta)\|^2\le \frac{\sigma^2}{M}.
\]
This is the central analytic reason the method can reduce communication frequency without sacrificing gradient-estimation quality: variance reduction comes from aggregating \(M\) asynchronously obtained samples, rather than from lock-step synchronization at the level of individual gradients [2509.24305].

## 4. Time model and convergence guarantees

The paper models the time to compute one stochastic gradient on agent \(i\) as
\[
h_i=\dot h_i \times H,
\]
where \(\dot h_i\) is the per-environment-step time and \(H\) is the trajectory length. Communication time for sending a vector is denoted by \(\kappa\), and the same parameter is used for centralized communication and decentralized collective operations such as AllReduce.

Under this model, the time to collect \(M\) gradients with Rennala’s asynchronous aggregator is bounded by
\[
O\!\left(\kappa+\min_{m\in[n]} \left[\left(\frac{1}{m}\sum_{i=1}^m \frac{1}{h_i}\right)^{-1}\!\left(\frac{M}{m}+1\right)\right]\right).
\]
The dependence is harmonic-mean-like: it allows the analysis to effectively ignore sufficiently slow workers by selecting the best subset size \(m\), which is the source of the method’s straggler robustness.

The main convergence theorem gives parameter choices for \(\eta\), \(\alpha\), \(H\), \(M_{\text{init}}\), and \(M\), and states that the method reaches an \(\varepsilon\)-stationary point after a number of global iterations with terms
\[
\frac{L_g\Delta}{\varepsilon^2},\qquad
\frac{\sqrt{L_h}\Delta}{\varepsilon^{3/2}},\qquad
\frac{\sigma^2}{M\varepsilon^2},\qquad
\frac{\sigma^2\sqrt{L_h}\Delta}{M\varepsilon^{7/2}},
\]
where \(\Delta:=J^*-J(\theta_0)\). The stated homogeneous total time bound is a \(\tilde{\mathcal O}(\cdot)\) expression with a communication component proportional to
\[
\kappa\left(\frac{L_g\Delta}{\varepsilon^2}+\frac{\sqrt{L_h}\Delta}{\varepsilon^{3/2}}\right)
\]
and a computation component proportional to
\[
\frac{1}{1-\gamma}\min_{m\in[n]} \left[ \left(\frac{1}{m}\sum_{i=1}^m \frac{1}{\dot h_i}\right)^{-1}(\cdots) \right],
\]
with the inner term including the variance-sensitive contributions \(\frac{\sigma^2}{m\varepsilon^2}\) and \(\frac{\sigma^2\sqrt{L_h}\Delta}{m\varepsilon^{7/2}}\). This separation makes explicit that one source of improvement comes from reducing the number of communication rounds, while another comes from a speed-aware aggregation rule that does not force every worker to determine each update time [2509.24305].

## 5. Comparison with distributed policy-gradient baselines

The paper positions Rennala NIGT against three baselines: synchronized Vanilla PG, synchronized NIGT, and AFedPG. The central point of comparison is not merely iteration complexity, but the joint computational and communication burden in asynchronous heterogeneous systems.

| Method | Characteristic complexity feature | Distinguishing limitation or advantage |
|---|---|---|
| Synchronized Vanilla PG | \(\max_i \dot h_i\left(\frac{1}{\varepsilon^2}+\frac{1}{n\varepsilon^4}\right)\) plus similar communication cost | Bottlenecked by the slowest worker |
| Synchronized NIGT | Communication cost \(\kappa\cdot \varepsilon^{-7/2}\) in the table | Less communication-efficient than Rennala NIGT |
| AFedPG | Communication \(O(\kappa\varepsilon^{-3})\), worst case \(O(\kappa\varepsilon^{-7/2})\) | Does not support AllReduce |
| Rennala NIGT | \(\min_{m\in[n]} \left[ \left(\frac{1}{m}\sum_{i=1}^m \frac{1}{\dot h_i}\right)^{-1} \left( \frac{1}{\varepsilon^2}+\frac{1}{m\varepsilon^{7/2}} \right) \right]\) | Supports AllReduce and improves communication complexity |

Against synchronized Vanilla PG, Rennala NIGT avoids explicit dependence on the slowest worker and replaces it with a subset-adaptive harmonic-mean expression. Against synchronized NIGT, the paper’s claim is primarily about communication efficiency: synchronized NIGT inherits better centralized optimization rates, but distributed RL still pays a communication cost \(\kappa\cdot \varepsilon^{-7/2}\) in the table, whereas Rennala NIGT reduces the communication dependence to \(O(\kappa\varepsilon^{-2})\) in the small-\(\varepsilon\) regime. Against AFedPG, the claimed advantages are twofold: communication improves from \(O(\kappa\varepsilon^{-3})\) to \(O(\kappa\varepsilon^{-2})\), and the computation term can decrease as additional workers are added instead of worsening with \(n\). The paper also emphasizes that Rennala NIGT supports AllReduce, while AFedPG does not [2509.24305].

## 6. Empirical behavior

The empirical study is conducted on MuJoCo tasks, with particular emphasis on Humanoid-v4, and also includes Reacher-v4, Walker2d-v4, and Hopper-v4. The reported qualitative pattern is systematic across regimes. When compute speeds are equal and there is no communication delay, Rennala NIGT, AFedPG, and synchronized NIGT behave similarly. Under heterogeneous computation times, Rennala NIGT converges faster than the baselines. When heterogeneity is combined with large communication delays, Rennala NIGT is reported as the most robust and the performance gap widens.

The paper also scales experiments to \(n=100\) agents. In that regime, equal-time settings again produce similar behavior across methods, while heterogeneous or communication-expensive settings preserve Rennala NIGT’s advantage. The strongest empirical gains are highlighted on Humanoid-v4 and Reacher-v4, where the method is often faster and its margin grows with systems heterogeneity.

The same paper additionally studies Malenia NIGT in a genuinely heterogeneous-environment experiment, where one agent sees normal Humanoid-v4 and another sees inverted states. That experiment is not evidence about Rennala NIGT itself, but it clarifies the boundary between the two algorithms: Rennala NIGT is the homogeneous-environment method, whereas Malenia NIGT is the heterogeneous extension [2509.24305].

## 7. Relation to broader asynchronous optimization literature

The name “Rennala” already appears in earlier distributed stochastic optimization. Rennala SGD is the optimal parallel stochastic gradient method for the homogeneous setup under a universal computation model that allows arbitrary worker-speed behavior, including changing speeds, outages, and heterogeneity. Its mechanism is a batching-and-waiting SGD scheme: all workers compute stochastic gradients at the current iterate, the method waits until a prescribed batch size is collected, averages the gradients, updates the model, and stops ongoing computations before the next round. In that setting, matching upper and lower bounds establish optimality up to constants [2408.04929].

Subsequent work narrows the scope of that optimality. In the presence of random compute times, especially heavy-tailed or failure-prone delays, MindFlayer SGD argues that Rennala SGD’s wait-for-batch strategy can perform arbitrarily badly, and proposes capped waiting and retry as a more robust alternative. The critique is specific to random-delay regimes and does not contradict deterministic-delay optimality; rather, it shows that fixed-time optimality does not automatically transfer to stochastic runtime models [2410.04285]. In parallel, work on asynchronous finite-sum optimization treats Rennala SGD as an important baseline but emphasizes that its \(\sigma^2\)-bounded variance assumption and small-\(\varepsilon\) dependence can be suboptimal for nonconvex finite-sum structure, motivating variance-reduced alternatives such as Freya PAGE [2405.15545].

This broader literature helps delimit Rennala NIGT precisely. Its guarantees concern homogeneous distributed RL with asynchronous computations and communication delays, not arbitrary heterogeneous environments and not random-delay models of the kind studied by MindFlayer. A common misconception is to equate “homogeneous” with full synchronization or equal worker speed. In the paper’s terminology, homogeneous refers only to shared environment, reward function, and policy class; heterogeneous compute speeds and nontrivial communication delays are explicit parts of the target setting, and the method’s asynchronous aggregation plus single-update communication pattern is designed around that systems model [2509.24305].

Source: https://www.emergentmind.com/topics/rennala-nigt