LSTM-ResB-PPO: DRL for Assistive Navigation

• The paper proposes LSTM-ResB-PPO to reweight and rank candidate trajectories within a two-stage momentum-constrained hybrid heuristic framework for assistive navigation.
• It employs residual blocks combined with LSTM temporal memory to enhance feature extraction, achieve better gradient flow, and converge 48% faster than a standard PPO baseline.
• The dual-stage cost modeling integrates Frenet-based heuristic sampling with DRL-based Cartesian refinement, resulting in improved safety, reduced cost variance, and higher reward stability.

Searching arXiv for the specified papers and closely related context. LSTM-ResB-PPO is a PPO-based residual actor-critic with temporal memory used as the second-stage learner in the momentum-constrained hybrid heuristic trajectory optimization framework (MHHTOF) for assistive navigation in visually impaired scenarios (Zeng et al., 19 Sep 2025). Within that framework, it does not replace heuristic trajectory generation; instead, it learns how to reweight and rank feasible candidate trajectories generated in the Frenet stage, using context-aware, interpretable, human-centered cost signals in Cartesian space (Zeng et al., 19 Sep 2025). The architecture combines residual blocks for stronger nonlinear representation and better gradient flow with an LSTM for temporal dependency modeling, and the reported experiments state that it converges faster than a PPO baseline while improving reward, cost, variance, and risk metrics (Zeng et al., 19 Sep 2025). Related recurrent reinforcement-learning work on partially observed settings argues that robustness depends critically on processing observation and action history as a single causal sequence, rather than privileging only the latest observation, which provides a useful contextual lens for interpreting the temporal component of LSTM-ResB-PPO (Omi et al., 2023).

1. Position within the MHHTOF pipeline

The framework in which LSTM-ResB-PPO is embedded is explicitly two-stage (Zeng et al., 19 Sep 2025). In the first stage, heuristic trajectory sampling cluster (HTSC) is generated in the Frenet coordinate system using polynomial parameterization and momentum-constrained feasibility considerations. This stage produces candidate trajectories that are smooth, feasible, and structurally aligned with road geometry (Zeng et al., 19 Sep 2025).

The Frenet representation is given by

$$\boldsymbol{x}(s(t), d(t)) = \boldsymbol{r}(s(t)) + d(t)\boldsymbol{n}_c(s(t))$$

with full Frenet state

$$\left[s, \dot{s}, \ddot{s}; d, \dot{d}, \ddot{d}, d', d''\right].$$

Candidate trajectories are generated with quintic polynomials,

$$\left\{ \begin{array}{l} s(t)=\sum_{i=0}^{5} a_{i} t^{i} \ d(s)=\sum_{i=0}^{5} b_{i} s^{i} \end{array} \right.$$

where the coefficients are solved from boundary conditions on position, velocity, and acceleration (Zeng et al., 19 Sep 2025).

The second stage performs residual-enhanced DRL selection and refinement in the Cartesian frame. This is the stage in which LSTM-ResB-PPO operates: it observes compact semantic and risk features, adaptively adjusts cost weights, evaluates candidate trajectories, and returns the trajectory minimizing the learned evaluation cost (Zeng et al., 19 Sep 2025). The paper frames this as a dual-stage cost modeling mechanism (DCMM), in which the Frenet stage handles structure, feasibility, and comfort via deterministic heuristic cost, and the Cartesian stage applies semantic, dynamic, and social refinement via DRL (Zeng et al., 19 Sep 2025).

This division of labor is central to the method’s interpretation. LSTM-ResB-PPO is not presented as a direct low-level controller; it is a policy-learning and trajectory-ranking back-end whose action space consists of adaptive cost weighting over already feasible candidates (Zeng et al., 19 Sep 2025). A plausible implication is that the method’s reported real-time feasibility is tied not only to network design but also to this restricted decision interface.

2. Network architecture and representational design

The proposed DRL model is a PPO-based actor-critic with two main augmentations: residual blocks and LSTM-based temporal modeling (Zeng et al., 19 Sep 2025). The paper names the architecture ResBlocksCustomPolicyRewardNet, used separately for the policy network and the value/reward network, with the same architecture skeleton but independent parameters (Zeng et al., 19 Sep 2025).

Each residual block is defined as

$\operatorname{ResBlock}(x) = f(x) + S(x)$

with shortcut path

$$S(x)= \begin{cases} x, & \text{if } \operatorname{dim}(x)=\operatorname{dim}(f(x)) \ W_s x, & \text{otherwise} \end{cases}$$

where $f(x)$ is a two-layer nonlinear transform described as linear projection plus ReLU, $S(x)$ is the skip connection, and $W_s$ is a learnable projection when dimensions mismatch (Zeng et al., 19 Sep 2025). The paper states that this design lets the network be deeper without vanishing gradients and without needing heavy normalization layers (Zeng et al., 19 Sep 2025).

After stacked residual blocks, the latent representation is fed into a lightweight LSTM,

$h_t = \operatorname{LSTM}(x_t^{\text{Res}, h_{t-1}, c_{t-1})$

followed by a linear output layer,

$\text{output} = \operatorname{Linear}(h_t).$

The LSTM hidden size is reported as 8, and the stated intent is to model sequential dependencies in high-level encoded features rather than raw inputs, thereby keeping inference efficient while still capturing temporal continuity in motion and reward evolution (Zeng et al., 19 Sep 2025).

The network layouts reported in the paper are as follows:

Branch	Layout	Output
Policy network	FlattenExtractor → Linear $$\left[s, \dot{s}, \ddot{s}; d, \dot{d}, \ddot{d}, d', d''\right].$$0 → ResBlock $$\left[s, \dot{s}, \ddot{s}; d, \dot{d}, \ddot{d}, d', d''\right].$$1 → ReLU → LSTM $$\left[s, \dot{s}, \ddot{s}; d, \dot{d}, \ddot{d}, d', d''\right].$$2	Linear $$\left[s, \dot{s}, \ddot{s}; d, \dot{d}, \ddot{d}, d', d''\right].$$3
Reward/value network	FlattenExtractor → Linear $$\left[s, \dot{s}, \ddot{s}; d, \dot{d}, \ddot{d}, d', d''\right].$$4 → ResBlock $$\left[s, \dot{s}, \ddot{s}; d, \dot{d}, \ddot{d}, d', d''\right].$$5 → ReLU → LSTM $$\left[s, \dot{s}, \ddot{s}; d, \dot{d}, \ddot{d}, d', d''\right].$$6	Linear $$\left[s, \dot{s}, \ddot{s}; d, \dot{d}, \ddot{d}, d', d''\right].$$7

These architectural details place LSTM-ResB-PPO within a broader family of recurrent actor-critic methods, but the specific combination of a compact residual MLP backbone with a small LSTM is distinctive in the paper’s presentation (Zeng et al., 19 Sep 2025). Related work on POMDPs emphasizes that temporal robustness improves when recurrent models encode a unified causal trajectory containing both observations and actions, rather than splitting “past” and “current” processing streams (Omi et al., 2023). This suggests a broader design principle: recurrent memory is most effective when aligned with the causal structure of sequential decision-making.

3. PPO basis and the specific role of residual and recurrent augmentation

The paper states that LSTM-ResB-PPO extends standard PPO, but it does not introduce a new mathematical PPO objective (Zeng et al., 19 Sep 2025). Instead, the method retains standard PPO optimization while changing the policy and value representations through residual-enhanced feature extraction, LSTM temporal memory, context-aware action parameterization as adaptive cost weights, and a hierarchical reward/cost structure (Zeng et al., 19 Sep 2025).

The standard PPO clipped surrogate objective is given as

$$\left[s, \dot{s}, \ddot{s}; d, \dot{d}, \ddot{d}, d', d''\right].$$8

with

$$\left[s, \dot{s}, \ddot{s}; d, \dot{d}, \ddot{d}, d', d''\right].$$9

and hyperparameters including clipping $$\left\{ \begin{array}{l} s(t)=\sum_{i=0}^{5} a_{i} t^{i} \ d(s)=\sum_{i=0}^{5} b_{i} s^{i} \end{array} \right.$$0, GAE $$\left\{ \begin{array}{l} s(t)=\sum_{i=0}^{5} a_{i} t^{i} \ d(s)=\sum_{i=0}^{5} b_{i} s^{i} \end{array} \right.$$1, learning rate $$\left\{ \begin{array}{l} s(t)=\sum_{i=0}^{5} a_{i} t^{i} \ d(s)=\sum_{i=0}^{5} b_{i} s^{i} \end{array} \right.$$2, and entropy coefficient $$\left\{ \begin{array}{l} s(t)=\sum_{i=0}^{5} a_{i} t^{i} \ d(s)=\sum_{i=0}^{5} b_{i} s^{i} \end{array} \right.$$3 (Zeng et al., 19 Sep 2025).

According to the paper, what is modified is not the PPO loss itself but the representational substrate on which PPO operates (Zeng et al., 19 Sep 2025). The residual blocks are intended to improve feature extraction and gradient flow, while the LSTM contributes temporal consistency and sequential dependence modeling (Zeng et al., 19 Sep 2025). The action space is not direct motion control; instead, it consists of adjustments to interpretable cost weights associated with trajectory evaluation (Zeng et al., 19 Sep 2025).

A useful contrast emerges when this is compared with recurrent TD3 work in POMDPs. That work explicitly notes that it does not use the PPO clipped objective and instead employs deterministic policy gradient with twin critics, target smoothing noise, and delayed policy updates (Omi et al., 2023). The comparison is relevant because it separates two dimensions that are often conflated: the choice of policy optimization algorithm and the choice of recurrent state-estimation architecture. LSTM-ResB-PPO belongs to the PPO family in optimization terms, whereas the POMDP study shows that temporal-memory design can be analyzed independently of PPO-specific machinery (Omi et al., 2023).

4. Dual-stage cost modeling and adaptive weight transfer

DCMM is central to the framework (Zeng et al., 19 Sep 2025). In the first stage, the Frenet-frame cost is

$$\left\{ \begin{array}{l} s(t)=\sum_{i=0}^{5} a_{i} t^{i} \ d(s)=\sum_{i=0}^{5} b_{i} s^{i} \end{array} \right.$$4

where $$\left\{ \begin{array}{l} s(t)=\sum_{i=0}^{5} a_{i} t^{i} \ d(s)=\sum_{i=0}^{5} b_{i} s^{i} \end{array} \right.$$5 captures lateral regularity, boundary adherence, and curvature smoothness, and $$\left\{ \begin{array}{l} s(t)=\sum_{i=0}^{5} a_{i} t^{i} \ d(s)=\sum_{i=0}^{5} b_{i} s^{i} \end{array} \right.$$6 captures longitudinal consistency, speed smoothness, and deceleration feasibility (Zeng et al., 19 Sep 2025). This stage creates the candidate set

$$\left\{ \begin{array}{l} s(t)=\sum_{i=0}^{5} a_{i} t^{i} \ d(s)=\sum_{i=0}^{5} b_{i} s^{i} \end{array} \right.$$7

In the second stage, each candidate is scored by

$$\left\{ \begin{array}{l} s(t)=\sum_{i=0}^{5} a_{i} t^{i} \ d(s)=\sum_{i=0}^{5} b_{i} s^{i} \end{array} \right.$$8

and the final chosen trajectory is

$$\left\{ \begin{array}{l} s(t)=\sum_{i=0}^{5} a_{i} t^{i} \ d(s)=\sum_{i=0}^{5} b_{i} s^{i} \end{array} \right.$$9

(Zeng et al., 19 Sep 2025).

The DRL policy does not directly command low-level motion. Instead, it adjusts the weights $\operatorname{ResBlock}(x) = f(x) + S(x)$0 attached to interpretable trajectory costs, using the update rule

$\operatorname{ResBlock}(x) = f(x) + S(x)$1

The paper describes this as the “weight transfer” mechanism: the Frenet stage produces feasible trajectories and implicit priorities, the Cartesian stage learns how to reweight those priorities based on context, and the weights remain interpretable and bounded (Zeng et al., 19 Sep 2025).

The candidate trajectories are screened first by feasibility checks and then by Cartesian cost evaluation (Zeng et al., 19 Sep 2025). The feasibility screening includes kinematic constraints, collision checks, and limits on acceleration, yaw rate, and curvature (Zeng et al., 19 Sep 2025). The Cartesian evaluation cost is written as

$\operatorname{ResBlock}(x) = f(x) + S(x)$2

with metrics

$\operatorname{ResBlock}(x) = f(x) + S(x)$3

(Zeng et al., 19 Sep 2025).

This cost-weighting formulation is one of the method’s defining characteristics. Rather than learning an opaque scalar score directly from trajectory candidates, the policy learns bounded adjustments to semantically labeled criteria. The paper characterizes this as human-centered and semantically aligned across both stages (Zeng et al., 19 Sep 2025).

5. Trajectory generation, momentum constraints, and planning semantics

The HTSC is built in the Frenet frame by varying endpoint states, fitting quintic polynomials, applying endpoint smoothing, and enforcing feasibility and momentum continuity (Zeng et al., 19 Sep 2025). For terminal smoothing, the paper gives a third-order state interpolation:

$\operatorname{ResBlock}(x) = f(x) + S(x)$4

with endpoint states at $\operatorname{ResBlock}(x) = f(x) + S(x)$5 and $\operatorname{ResBlock}(x) = f(x) + S(x)$6 solved analytically (Zeng et al., 19 Sep 2025).

The momentum-constrained trajectory optimization (MTO) formulation models the trajectory as a constrained Lagrangian system over the Frenet state

$\operatorname{ResBlock}(x) = f(x) + S(x)$7

with system form

$\operatorname{ResBlock}(x) = f(x) + S(x)$8

where

$\operatorname{ResBlock}(x) = f(x) + S(x)$9

(Zeng et al., 19 Sep 2025).

The optimization objective is

$$S(x)= \begin{cases} x, & \text{if } \operatorname{dim}(x)=\operatorname{dim}(f(x)) \ W_s x, & \text{otherwise} \end{cases}$$0

and the uncertainty-aware cost includes

$$S(x)= \begin{cases} x, & \text{if } \operatorname{dim}(x)=\operatorname{dim}(f(x)) \ W_s x, & \text{otherwise} \end{cases}$$1

with

$$S(x)= \begin{cases} x, & \text{if } \operatorname{dim}(x)=\operatorname{dim}(f(x)) \ W_s x, & \text{otherwise} \end{cases}$$2

The expanded Lagrangian is

$$S(x)= \begin{cases} x, & \text{if } \operatorname{dim}(x)=\operatorname{dim}(f(x)) \ W_s x, & \text{otherwise} \end{cases}$$3

(Zeng et al., 19 Sep 2025).

The force decomposition includes

$$S(x)= \begin{cases} x, & \text{if } \operatorname{dim}(x)=\operatorname{dim}(f(x)) \ W_s x, & \text{otherwise} \end{cases}$$4

and the crowd repulsion is anisotropic,

$$S(x)= \begin{cases} x, & \text{if } \operatorname{dim}(x)=\operatorname{dim}(f(x)) \ W_s x, & \text{otherwise} \end{cases}$$5

(Zeng et al., 19 Sep 2025).

The paper interprets these terms as kinetic motion energy, a guidance force term for assistive cues, crowd repulsion for social compliance and collision avoidance, a smoothness regularizer $$S(x)= \begin{cases} x, & \text{if } \operatorname{dim}(x)=\operatorname{dim}(f(x)) \ W_s x, & \text{otherwise} \end{cases}$$6, and a perceptual uncertainty penalty $$S(x)= \begin{cases} x, & \text{if } \operatorname{dim}(x)=\operatorname{dim}(f(x)) \ W_s x, & \text{otherwise} \end{cases}$$7 (Zeng et al., 19 Sep 2025). This is the “momentum-constrained” aspect: trajectories are regularized not just by geometry but by force-like terms that encourage smooth, safe, socially aware motion (Zeng et al., 19 Sep 2025). In this context, LSTM-ResB-PPO functions as an adaptive evaluator over trajectories whose semantics have already been structured by the MTO formulation.

6. Experimental configuration and reported quantitative results

The experiments are conducted on the CommonRoad benchmark using Gymnasium, Stable-Baselines3, and PyTorch (Zeng et al., 19 Sep 2025). The reported hardware consists of 4 × NVIDIA GeForce RTX 4090 GPUs (24GB each), 128GB DDR5 RAM, and a 2TB NVMe SSD (Zeng et al., 19 Sep 2025). Training uses more than $$S(x)= \begin{cases} x, & \text{if } \operatorname{dim}(x)=\operatorname{dim}(f(x)) \ W_s x, & \text{otherwise} \end{cases}$$8 time steps; the best model is selected around $$S(x)= \begin{cases} x, & \text{if } \operatorname{dim}(x)=\operatorname{dim}(f(x)) \ W_s x, & \text{otherwise} \end{cases}$$9 steps; checkpoints are saved every 40,000 steps; and early stopping is used if there is no significant improvement in 10 evaluations (Zeng et al., 19 Sep 2025).

The reported hyperparameters are

$f(x)$0

with the explicit note that the table in the paper also shows $f(x)$1, while the text explicitly states $f(x)$2, creating an inconsistency in the paper (Zeng et al., 19 Sep 2025).

The quasi-ablation compares PPO baseline, LSTM-ResB-PPO, Bi-LSTM-ResB-PPO, RNN-ResB-PPO, Bi-RNN-ResB-PPO, GRU-ResB-PPO, and Bi-GRU-ResB-PPO (Zeng et al., 19 Sep 2025). The key findings reported are that LSTM-ResB-PPO converges at approximately 520,000 steps, whereas baseline PPO converges at 1,000,000 steps, making LSTM-ResB-PPO about 48% faster (Zeng et al., 19 Sep 2025). Final reward is reported as 861.138 for LSTM-ResB-PPO and 852.301 for PPO; peak reward as 877.502 for LSTM-ResB-PPO and 869.986 for PPO (Zeng et al., 19 Sep 2025). Reward stability is summarized by noting that LSTM-ResB-PPO has only 1 fluctuation, while GRU/Bi-GRU are much less stable and RNN-based variants often fail to converge meaningfully (Zeng et al., 19 Sep 2025).

The paper’s reported comparison table is:

Algorithm	CS	FP	CP	PR	MR	NF
PPO baseline	1,000,000	822.301	855.022	869.986	841.548	0
LSTM-ResB-PPO	520,000	861.138	861.055	877.502	851.776	1
Bi-LSTM-ResB-PPO	1,560,000	-2541.737	893.035	895.653	889.659	0
RNN-ResB-PPO	960,000	-4683.867	-4683.656	-4680.461	-4683.867	0
Bi-RNN-ResB-PPO	1,520,000	-4695.173	-4802.599	-4782.936	-4815.568	2
GRU-ResB-PPO	–	185.280	–	–	–	5
Bi-GRU-ResB-PPO	–	-4687.911	–	–	–	7

The model also converges to a stable episode length of 147 in about 520,000 steps, while baseline PPO also ends at 147 but more slowly (Zeng et al., 19 Sep 2025).

For the best models, the reported cost and safety metrics are:

Algorithm	Avg Cost Mean	Avg Cost Var	Avg Ego Risk	Avg Obstacle Risk
PPO baseline	0.166472	0.102921	$f(x)$3	$f(x)$4
LSTM-ResB-PPO	0.116065	0.048067	$f(x)$5	$f(x)$6

The reported improvements are cost mean down by 30.3%, cost variance down by 53.3%, ego risk down by 77.2%, and obstacle risk down by more than 80% (Zeng et al., 19 Sep 2025).

Scenario-level comparison is also reported:

Scenario	Algorithm	Goal reached	# trajectory clusters	Avg ego-risk	Avg costs
DEU_Lengede-21_1_T-15	PPO	0	152	$f(x)$7	6.6384
DEU_Lengede-21_1_T-15	LSTM-ResB-PPO	1	150	$f(x)$8	5.4286
ZAM_Junction-1_119_T-1	PPO	1	146	$f(x)$9	53.015
ZAM_Junction-1_119_T-1	LSTM-ResB-PPO	1	146	$S(x)$0	36.3689
USA_Tanker-1_7_T-1	PPO	1	13	$S(x)$1	31.5190
USA_Tanker-1_7_T-1	LSTM-ResB-PPO	1	13	$S(x)$2	30.6837

The paper summarizes these results as showing that the proposed model reaches the goal consistently, lowers cost, does not increase trajectory cluster count, and improves or matches safety (Zeng et al., 19 Sep 2025).

7. Interpretation, safety claims, and relation to recurrent RL in POMDPs

The paper’s main claims are that LSTM-ResB-PPO makes assistive navigation safer, more stable, and more real-time feasible (Zeng et al., 19 Sep 2025). Safety is associated with lower ego-risk and obstacle-risk, better lane adherence, fewer dangerous deviations, and better handling of dynamic obstacles and oncoming traffic (Zeng et al., 19 Sep 2025). Stability is associated with smoother speed and acceleration profiles, fewer reward oscillations, lower cost variance, better temporal consistency due to LSTM, and better gradient flow and training stability due to residual blocks (Zeng et al., 19 Sep 2025). Real-time feasibility is attributed to a lightweight architecture, an LSTM hidden size of only 8, a compact residual MLP backbone, and a policy that acts on cost weights rather than directly producing complex control outputs (Zeng et al., 19 Sep 2025).

The same paper repeatedly emphasizes comfort, interpretability, safety margins, human-centered semantic costs, perceptual uncertainty handling, and conservative, smooth motion generation aligned with limited perception (Zeng et al., 19 Sep 2025). In the DEU_Lengede case, it reports that the baseline fails and drifts into oncoming traffic, whereas LSTM-ResB-PPO stays in the correct lane; in ZAM_Tjunction, both succeed, but the proposed method shows much smoother deceleration and less acceleration spiking (Zeng et al., 19 Sep 2025).

A common source of confusion is the relation between LSTM-ResB-PPO and recurrent RL work that is not PPO-based. The paper on dynamic deep reinforcement learning in POMDPs does not propose “LSTM-ResB-PPO” by name and is based on TD3 rather than PPO (Omi et al., 2023). Its relevant contribution is the argument that, in POMDPs, the policy should consume observation and action history together, and that the recurrent encoder should process the entire recent trajectory as one sequence (Omi et al., 2023). It further states that any LSTM-based method beats plain TD3 in POMDP settings, that adding action sequences improves performance over using observation sequences alone, and that modified one-stream architectures are generally the most robust (Omi et al., 2023).

This comparison matters because LSTM-ResB-PPO is frequently interpretable along two axes: PPO as the optimization framework, and LSTM/residual design as the representational framework. The TD3-based POMDP study indicates that recurrent robustness is tightly coupled to the causal content of the encoded trajectory, especially the inclusion of past actions (Omi et al., 2023). LSTM-ResB-PPO, by contrast, is described in terms of temporal modeling over high-level encoded features and adaptive cost-weight selection within a two-stage planner (Zeng et al., 19 Sep 2025). A plausible implication is that both works share an interest in temporal state reconstruction, but they instantiate that interest at different points in the decision stack and under different optimization regimes.

Overall, LSTM-ResB-PPO denotes a PPO-based residual actor-critic with LSTM temporal memory, deployed as the second-stage learner in MHHTOF to reweight and rank feasible trajectories rather than to generate them from scratch (Zeng et al., 19 Sep 2025). Its reported significance lies in the combination of residual feature extraction, compact sequential memory, interpretable cost-weight adaptation, and integration with a momentum-constrained heuristic planning front-end (Zeng et al., 19 Sep 2025).