---
title: 'LSTM-ResB-PPO: DRL for Assistive Navigation'
url: https://www.emergentmind.com/topics/lstm-resb-ppo
type: topic
---

# LSTM-ResB-PPO: DRL for Assistive Navigation

Searching arXiv for the specified papers and closely related context.
LSTM-ResB-PPO is a PPO-based residual actor-critic with temporal memory used as the second-stage learner in the momentum-constrained hybrid heuristic trajectory optimization framework (MHHTOF) for assistive navigation in visually impaired scenarios [2509.15582]. Within that framework, it does not replace heuristic trajectory generation; instead, it learns how to reweight and rank feasible candidate trajectories generated in the Frenet stage, using context-aware, interpretable, human-centered cost signals in Cartesian space [2509.15582]. The architecture combines residual blocks for stronger nonlinear representation and better gradient flow with an LSTM for temporal dependency modeling, and the reported experiments state that it converges faster than a PPO baseline while improving reward, cost, variance, and risk metrics [2509.15582]. Related recurrent reinforcement-learning work on partially observed settings argues that robustness depends critically on processing observation and action history as a single causal sequence, rather than privileging only the latest observation, which provides a useful contextual lens for interpreting the temporal component of LSTM-ResB-PPO [2307.15931].

## 1. Position within the MHHTOF pipeline

The framework in which LSTM-ResB-PPO is embedded is explicitly two-stage [2509.15582]. In the first stage, heuristic trajectory sampling cluster (HTSC) is generated in the Frenet coordinate system using polynomial parameterization and momentum-constrained feasibility considerations. This stage produces candidate trajectories that are smooth, feasible, and structurally aligned with road geometry [2509.15582].

The Frenet representation is given by
$$
\boldsymbol{x}(s(t), d(t)) = \boldsymbol{r}(s(t)) + d(t)\boldsymbol{n}_c(s(t))
$$
with full Frenet state
$$
\left[s, \dot{s}, \ddot{s}; d, \dot{d}, \ddot{d}, d', d''\right].
$$
Candidate trajectories are generated with quintic polynomials,
$$
\left\{ \begin{array}{l}
s(t)=\sum_{i=0}^{5} a_{i} t^{i} \\
d(s)=\sum_{i=0}^{5} b_{i} s^{i}
\end{array} \right.
$$
where the coefficients are solved from boundary conditions on position, velocity, and acceleration [2509.15582].

The second stage performs residual-enhanced DRL selection and refinement in the Cartesian frame. This is the stage in which LSTM-ResB-PPO operates: it observes compact semantic and risk features, adaptively adjusts cost weights, evaluates candidate trajectories, and returns the trajectory minimizing the learned evaluation cost [2509.15582]. The paper frames this as a dual-stage cost modeling mechanism (DCMM), in which the Frenet stage handles structure, feasibility, and comfort via deterministic heuristic cost, and the Cartesian stage applies semantic, dynamic, and social refinement via DRL [2509.15582].

This division of labor is central to the method’s interpretation. LSTM-ResB-PPO is not presented as a direct low-level controller; it is a policy-learning and trajectory-ranking back-end whose action space consists of adaptive cost weighting over already feasible candidates [2509.15582]. A plausible implication is that the method’s reported real-time feasibility is tied not only to network design but also to this restricted decision interface.

## 2. Network architecture and representational design

The proposed DRL model is a PPO-based actor-critic with two main augmentations: residual blocks and LSTM-based temporal modeling [2509.15582]. The paper names the architecture **ResBlocksCustomPolicyRewardNet**, used separately for the policy network and the value/reward network, with the same architecture skeleton but independent parameters [2509.15582].

Each residual block is defined as
$$
\operatorname{ResBlock}(x) = f(x) + S(x)
$$
with shortcut path
$$
S(x)= \begin{cases}
x, & \text{if } \operatorname{dim}(x)=\operatorname{dim}(f(x)) \\
W_s x, & \text{otherwise}
\end{cases}
$$
where \(f(x)\) is a two-layer nonlinear transform described as linear projection plus ReLU, \(S(x)\) is the skip connection, and \(W_s\) is a learnable projection when dimensions mismatch [2509.15582]. The paper states that this design lets the network be deeper without vanishing gradients and without needing heavy normalization layers [2509.15582].

After stacked residual blocks, the latent representation is fed into a lightweight LSTM,
$$
h_t = \operatorname{LSTM}(x_t^{\text{Res}, h_{t-1}, c_{t-1})
$$
followed by a linear output layer,
$$
\text{output} = \operatorname{Linear}(h_t).
$$
The LSTM hidden size is reported as 8, and the stated intent is to model sequential dependencies in high-level encoded features rather than raw inputs, thereby keeping inference efficient while still capturing temporal continuity in motion and reward evolution [2509.15582].

The network layouts reported in the paper are as follows:

| Branch | Layout | Output |
|---|---|---|
| Policy network | FlattenExtractor → Linear \(4 \to 32\) → ResBlock \(\times 2\) → ReLU → LSTM \(100 \to 8\) | Linear \(8 \to 2\) |
| Reward/value network | FlattenExtractor → Linear \(4 \to 32\) → ResBlock \(\times 2\) → ReLU → LSTM \(100 \to 8\) | Linear \(8 \to 1\) |

These architectural details place LSTM-ResB-PPO within a broader family of recurrent actor-critic methods, but the specific combination of a compact residual MLP backbone with a small LSTM is distinctive in the paper’s presentation [2509.15582]. Related work on POMDPs emphasizes that temporal robustness improves when recurrent models encode a unified causal trajectory containing both observations and actions, rather than splitting “past” and “current” processing streams [2307.15931]. This suggests a broader design principle: recurrent memory is most effective when aligned with the causal structure of sequential decision-making.

## 3. PPO basis and the specific role of residual and recurrent augmentation

The paper states that LSTM-ResB-PPO extends standard PPO, but it does not introduce a new mathematical PPO objective [2509.15582]. Instead, the method retains standard PPO optimization while changing the policy and value representations through residual-enhanced feature extraction, LSTM temporal memory, context-aware action parameterization as adaptive cost weights, and a hierarchical reward/cost structure [2509.15582].

The standard PPO clipped surrogate objective is given as
$$
L^{\text{PPO}(\theta)= \mathbb{E}_t\left[ \min\left( r_t(\theta)\hat{A}_t,\, \operatorname{clip}(r_t(\theta),1-\epsilon,1+\epsilon)\hat{A}_t \right) \right]
$$
with
$$
r_t(\theta)=\frac{\pi_\theta(a_t|o_t)}{\pi_{\theta_{\text{old}(a_t|o_t)}
$$
and hyperparameters including clipping \(\epsilon = 0.1\), GAE \(\lambda = 0.97\), learning rate \(3\times 10^{-4}\), and entropy coefficient \(c_2 = 0.01\) [2509.15582].

According to the paper, what is modified is not the PPO loss itself but the representational substrate on which PPO operates [2509.15582]. The residual blocks are intended to improve feature extraction and gradient flow, while the LSTM contributes temporal consistency and sequential dependence modeling [2509.15582]. The action space is not direct motion control; instead, it consists of adjustments to interpretable cost weights associated with trajectory evaluation [2509.15582].

A useful contrast emerges when this is compared with recurrent TD3 work in POMDPs. That work explicitly notes that it does **not** use the PPO clipped objective and instead employs deterministic policy gradient with twin critics, target smoothing noise, and delayed policy updates [2307.15931]. The comparison is relevant because it separates two dimensions that are often conflated: the choice of policy optimization algorithm and the choice of recurrent state-estimation architecture. LSTM-ResB-PPO belongs to the PPO family in optimization terms, whereas the POMDP study shows that temporal-memory design can be analyzed independently of PPO-specific machinery [2307.15931].

## 4. Dual-stage cost modeling and adaptive weight transfer

DCMM is central to the framework [2509.15582]. In the first stage, the Frenet-frame cost is
$$
J[d,s] = J_d[d] + k_s J_s[s], \qquad k_s > 0
$$
where \(J_d[d]\) captures lateral regularity, boundary adherence, and curvature smoothness, and \(J_s[s]\) captures longitudinal consistency, speed smoothness, and deceleration feasibility [2509.15582]. This stage creates the candidate set
$$
\mathcal{T}=\{x^{(1)}(t),x^{(2)}(t),\dots,x^{(N)}(t)\}.
$$

In the second stage, each candidate is scored by
$$
J_{\text{eval}^{\text{DRL}(x^{(i)}(t))= \sum_{j=1}^{n}\lambda_j^{\text{DRL}\cdot \phi_j(x^{(i)}(t)), \qquad x^{(i)}(t)\in\mathcal{T}
$$
and the final chosen trajectory is
$$
x^*(t)=\arg\min_{x^{(i)}(t)\in\mathcal{T} J_{\text{eval}^{\text{DRL}(x^{(i)}(t))
$$
[2509.15582].

The DRL policy does not directly command low-level motion. Instead, it adjusts the weights \(\lambda_i^{\text{DRL}}\) attached to interpretable trajectory costs, using the update rule
$$
\lambda_i^t = \text{clip}\left(\lambda_i^{t-1} + \Delta\lambda_i^{\text{DRL}, \lambda_i^{\min}, \lambda_i^{\max}\right).
$$
The paper describes this as the “weight transfer” mechanism: the Frenet stage produces feasible trajectories and implicit priorities, the Cartesian stage learns how to reweight those priorities based on context, and the weights remain interpretable and bounded [2509.15582].

The candidate trajectories are screened first by feasibility checks and then by Cartesian cost evaluation [2509.15582]. The feasibility screening includes kinematic constraints, collision checks, and limits on acceleration, yaw rate, and curvature [2509.15582]. The Cartesian evaluation cost is written as
$$
J_s = \sum_{i=1}^{n}\lambda_i^{\text{DRL}\cdot \phi_i(x(t))
$$
with metrics
$$
\left\{ \begin{array}{ll}
\phi_1(x)=\int a_x^2(t)\,dt & \text{(AE)}\\
\phi_2(x)=\int j^2(t)\,dt & \text{(JM)}\\
\phi_3(x)=\int |v_x(t)-v_{\text{desire}(t)|\,dt + (v_x(t_f)-v_{\text{desire}(t_f))^2 & \text{(VO)}\\
\phi_4(x)=\int d^2(t)\,dt & \text{(PD)}\\
\phi_5(x)=\int \frac{1}{\Delta x_{\text{obs}^2(t)}\,dt & \text{(OPP)}\\
\phi_6(x)=\int f(x,0,\Sigma_{\text{rot})\,dx\,dt & \text{(RFP)}\\
\phi_7(x)=\int \frac{1-t/T}{d_{\mathrm{M}(v_j(t),v_x(t),\Sigma(t))}\,dt & \text{(SC)}
\end{array} \right.
$$
[2509.15582].

This cost-weighting formulation is one of the method’s defining characteristics. Rather than learning an opaque scalar score directly from trajectory candidates, the policy learns bounded adjustments to semantically labeled criteria. The paper characterizes this as human-centered and semantically aligned across both stages [2509.15582].

## 5. Trajectory generation, momentum constraints, and planning semantics

The HTSC is built in the Frenet frame by varying endpoint states, fitting quintic polynomials, applying endpoint smoothing, and enforcing feasibility and momentum continuity [2509.15582]. For terminal smoothing, the paper gives a third-order state interpolation:
$$
\left\{ \begin{array}{l}
\xi_{i_1}(t)=\sum_{j=0}^{5} c_j t^j \\
\xi_{i_2}(t)=\dot{\xi}_{i_1}(t)=\sum_{j=1}^{5} j c_j t^{j-1} \\
\xi_{i_3}(t)=\ddot{\xi}_{i_1}(t)=\sum_{j=2}^{5} j(j-1)c_j t^{j-2}
\end{array} \right.
$$
with endpoint states at \(t=0\) and \(t=\tau\) solved analytically [2509.15582].

The momentum-constrained trajectory optimization (MTO) formulation models the trajectory as a constrained Lagrangian system over the Frenet state
$$
\boldsymbol{\xi}(t)=\left[s(t),\dot{s}(t),\ddot{s}(t),d(t),\dot{d}(t),\ddot{d}(t)\right]
$$
with system form
$$
\dot{\boldsymbol{\xi}(t)= \begin{bmatrix} A & \mathbf{0}\\ \mathbf{0} & A \end{bmatrix}\boldsymbol{\xi}(t)+ \begin{bmatrix} B & \mathbf{0}\\ \mathbf{0} & B \end{bmatrix}\boldsymbol{u}(t)
$$
where
$$
A=\begin{bmatrix} 0&1&0\\ 0&0&1\\ 0&0&0 \end{bmatrix},\qquad B=\begin{bmatrix} 0\\0\\1 \end{bmatrix}
$$
[2509.15582].

The optimization objective is
$$
S=\lim_{N\to\infty}\frac{1}{N}\sum_{k=0}^{N-1}J(\boldsymbol{u}(t_k),\boldsymbol{\xi}(t_k),\Phi),
$$
and the uncertainty-aware cost includes
$$
S_{\text{uncertainty} = (t_\tau - t_0)\cdot \mathbb{E}_{t\in[t_0,t_\tau]}[L_i(t)]
$$
with
$$
L_i(t)=E_{\text{motion},i}(t)-E_{\text{guidance},i}(t).
$$
The expanded Lagrangian is
$$
L_i(t)= \frac{1}{2}m_i\|v_i(t)\|^2 - F_{\text{guidance}(t)}\cdot v_i(t) - F_{\text{crowd}(t)}\cdot (v_i(t)-v_j(t)) + \lambda_s\|\dot{v}_i(t)\|^2 + \lambda_u \operatorname{Tr}\left[\Sigma_i^{\text{perception}(t)}\right]
$$
[2509.15582].

The force decomposition includes
$$
\boldsymbol{F}_{\text{guidance}(t)}= \begin{bmatrix} - G_{i,s}(t)\\ F_{\mathrm{li}(t)} - G_{i,d}(t) \end{bmatrix}
$$
and the crowd repulsion is anisotropic,
$$
\boldsymbol{F}_{\text{crowd}^{i,j}(t)= \frac{1}{2}m_j\|v_j(t)\|^2 \cdot r_{ij}(t) \left(\frac{1}{r_s^2(t)}-\frac{1}{r_d^2(t)}\right)\mathbf{n}_{ij}(t)
$$
[2509.15582].

The paper interprets these terms as kinetic motion energy, a guidance force term for assistive cues, crowd repulsion for social compliance and collision avoidance, a smoothness regularizer \(\lambda_s\), and a perceptual uncertainty penalty \(\lambda_u\) [2509.15582]. This is the “momentum-constrained” aspect: trajectories are regularized not just by geometry but by force-like terms that encourage smooth, safe, socially aware motion [2509.15582]. In this context, LSTM-ResB-PPO functions as an adaptive evaluator over trajectories whose semantics have already been structured by the MTO formulation.

## 6. Experimental configuration and reported quantitative results

The experiments are conducted on the CommonRoad benchmark using Gymnasium, Stable-Baselines3, and PyTorch [2509.15582]. The reported hardware consists of 4 × NVIDIA GeForce RTX 4090 GPUs (24GB each), 128GB DDR5 RAM, and a 2TB NVMe SSD [2509.15582]. Training uses more than \(10^6\) time steps; the best model is selected around \(5\times10^5\) steps; checkpoints are saved every 40,000 steps; and early stopping is used if there is no significant improvement in 10 evaluations [2509.15582].

The reported hyperparameters are
$$
l=3\times 10^{-4},\quad \epsilon=0.1,\quad \lambda=0.97,\quad \gamma=0.97,\quad B=2352,\quad N^E=5,\quad c_2=0.01
$$
with the explicit note that the table in the paper also shows \(\gamma=0.80\), while the text explicitly states \(\gamma=0.97\), creating an inconsistency in the paper [2509.15582].

The quasi-ablation compares PPO baseline, LSTM-ResB-PPO, Bi-LSTM-ResB-PPO, RNN-ResB-PPO, Bi-RNN-ResB-PPO, GRU-ResB-PPO, and Bi-GRU-ResB-PPO [2509.15582]. The key findings reported are that LSTM-ResB-PPO converges at approximately 520,000 steps, whereas baseline PPO converges at 1,000,000 steps, making LSTM-ResB-PPO about 48% faster [2509.15582]. Final reward is reported as 861.138 for LSTM-ResB-PPO and 852.301 for PPO; peak reward as 877.502 for LSTM-ResB-PPO and 869.986 for PPO [2509.15582]. Reward stability is summarized by noting that LSTM-ResB-PPO has only 1 fluctuation, while GRU/Bi-GRU are much less stable and RNN-based variants often fail to converge meaningfully [2509.15582].

The paper’s reported comparison table is:

| Algorithm | CS | FP | CP | PR | MR | NF |
|---|---:|---:|---:|---:|---:|---:|
| PPO baseline | 1,000,000 | 822.301 | 855.022 | 869.986 | 841.548 | 0 |
| LSTM-ResB-PPO | 520,000 | 861.138 | 861.055 | 877.502 | 851.776 | 1 |
| Bi-LSTM-ResB-PPO | 1,560,000 | -2541.737 | 893.035 | 895.653 | 889.659 | 0 |
| RNN-ResB-PPO | 960,000 | -4683.867 | -4683.656 | -4680.461 | -4683.867 | 0 |
| Bi-RNN-ResB-PPO | 1,520,000 | -4695.173 | -4802.599 | -4782.936 | -4815.568 | 2 |
| GRU-ResB-PPO | – | 185.280 | – | – | – | 5 |
| Bi-GRU-ResB-PPO | – | -4687.911 | – | – | – | 7 |

The model also converges to a stable episode length of 147 in about 520,000 steps, while baseline PPO also ends at 147 but more slowly [2509.15582].

For the best models, the reported cost and safety metrics are:

| Algorithm | Avg Cost Mean | Avg Cost Var | Avg Ego Risk | Avg Obstacle Risk |
|---|---:|---:|---:|---:|
| PPO baseline | 0.166472 | 0.102921 | \(4.848913\times10^{-4}\) | \(1.716621\times10^{-4}\) |
| LSTM-ResB-PPO | 0.116065 | 0.048067 | \(1.053492\times10^{-4}\) | \(3.336874\times10^{-5}\) |

The reported improvements are cost mean down by 30.3%, cost variance down by 53.3%, ego risk down by 77.2%, and obstacle risk down by more than 80% [2509.15582].

Scenario-level comparison is also reported:

| Scenario | Algorithm | Goal reached | # trajectory clusters | Avg ego-risk | Avg costs |
|---|---|---:|---:|---:|---:|
| DEU_Lengede-21_1_T-15 | PPO | 0 | 152 | \(2.25395e{-14}\) | 6.6384 |
| DEU_Lengede-21_1_T-15 | LSTM-ResB-PPO | 1 | 150 | \(8.4692e{-16}\) | 5.4286 |
| ZAM_Junction-1_119_T-1 | PPO | 1 | 146 | \(9.7826e{-5}\) | 53.015 |
| ZAM_Junction-1_119_T-1 | LSTM-ResB-PPO | 1 | 146 | \(8.9332e{-5}\) | 36.3689 |
| USA_Tanker-1_7_T-1 | PPO | 1 | 13 | \(3.2864e{-5}\) | 31.5190 |
| USA_Tanker-1_7_T-1 | LSTM-ResB-PPO | 1 | 13 | \(3.2864e{-5}\) | 30.6837 |

The paper summarizes these results as showing that the proposed model reaches the goal consistently, lowers cost, does not increase trajectory cluster count, and improves or matches safety [2509.15582].

## 7. Interpretation, safety claims, and relation to recurrent RL in POMDPs

The paper’s main claims are that LSTM-ResB-PPO makes assistive navigation safer, more stable, and more real-time feasible [2509.15582]. Safety is associated with lower ego-risk and obstacle-risk, better lane adherence, fewer dangerous deviations, and better handling of dynamic obstacles and oncoming traffic [2509.15582]. Stability is associated with smoother speed and acceleration profiles, fewer reward oscillations, lower cost variance, better temporal consistency due to LSTM, and better gradient flow and training stability due to residual blocks [2509.15582]. Real-time feasibility is attributed to a lightweight architecture, an LSTM hidden size of only 8, a compact residual MLP backbone, and a policy that acts on cost weights rather than directly producing complex control outputs [2509.15582].

The same paper repeatedly emphasizes comfort, interpretability, safety margins, human-centered semantic costs, perceptual uncertainty handling, and conservative, smooth motion generation aligned with limited perception [2509.15582]. In the DEU_Lengede case, it reports that the baseline fails and drifts into oncoming traffic, whereas LSTM-ResB-PPO stays in the correct lane; in ZAM_Tjunction, both succeed, but the proposed method shows much smoother deceleration and less acceleration spiking [2509.15582].

A common source of confusion is the relation between LSTM-ResB-PPO and recurrent RL work that is not PPO-based. The paper on dynamic deep reinforcement learning in POMDPs does not propose “LSTM-ResB-PPO” by name and is based on TD3 rather than PPO [2307.15931]. Its relevant contribution is the argument that, in POMDPs, the policy should consume observation and action history together, and that the recurrent encoder should process the entire recent trajectory as one sequence [2307.15931]. It further states that any LSTM-based method beats plain TD3 in POMDP settings, that adding action sequences improves performance over using observation sequences alone, and that modified one-stream architectures are generally the most robust [2307.15931].

This comparison matters because LSTM-ResB-PPO is frequently interpretable along two axes: PPO as the optimization framework, and LSTM/residual design as the representational framework. The TD3-based POMDP study indicates that recurrent robustness is tightly coupled to the causal content of the encoded trajectory, especially the inclusion of past actions [2307.15931]. LSTM-ResB-PPO, by contrast, is described in terms of temporal modeling over high-level encoded features and adaptive cost-weight selection within a two-stage planner [2509.15582]. A plausible implication is that both works share an interest in temporal state reconstruction, but they instantiate that interest at different points in the decision stack and under different optimization regimes.

Overall, LSTM-ResB-PPO denotes a PPO-based residual actor-critic with LSTM temporal memory, deployed as the second-stage learner in MHHTOF to reweight and rank feasible trajectories rather than to generate them from scratch [2509.15582]. Its reported significance lies in the combination of residual feature extraction, compact sequential memory, interpretable cost-weight adaptation, and integration with a momentum-constrained heuristic planning front-end [2509.15582].

Source: https://www.emergentmind.com/topics/lstm-resb-ppo