Object-Sequenced LSTM for 3D Detection

• Object-Sequenced LSTM is a deep learning architecture that leverages sequential LiDAR data to achieve robust temporal 3D object detection in autonomous driving.
• It integrates a sparse 3D convolutional LSTM module with a U-Net backbone, efficiently fusing per-frame features, memory states, and temporal consistency.
• The method outperforms traditional approaches by improving mAP by up to 7.5% while ensuring real-time performance with low computational overhead.

An Object-Sequenced LSTM is a deep neural architecture designed for temporal 3D object detection in LiDAR point clouds, particularly in autonomous driving contexts. It robustly leverages sequential LiDAR data by integrating features, memory states, and temporal consistency, employing a sparse 3D convolutional LSTM module with a U-Net backbone. This approach outperforms traditional frame-wise and early-fusion methods, improving multi-frame 3D object detection metrics while maintaining computational efficiency (Huang et al., 2020).

1. End-to-End Architecture and Data Flow

The Object-Sequenced LSTM employs a pipeline comprising six stages at each temporal step $t$:

1. Input:
   • Raw LiDAR sweep $P_t \in \mathbb{R}^{N\times 3}$ with $N \approx 1.8\times 10^5$ points.
   • Memory states from the previous frame: hidden features $H_{t-1}$ and cell features $C_{t-1}$ for $N' \approx 3\times 10^4$ “high-score” points.
2. Per-frame SparseConv U-Net Backbone:
   • Voxelizes $P_t$ with a grid size of $0.2~\text{m}^3$.
   • Encoder: 6 blocks of $3\times 3\times 3$ sparse 3D convs and max-pooling, with dimension progression $[64, 96, 128, 160, 192, 224, 256]$.
   • Decoder: symmetric structure with upsampling and skip-connections.
   • De-voxelizes to obtain per-point features $P_t \in \mathbb{R}^{N\times 3}$0, $P_t \in \mathbb{R}^{N\times 3}$1.
3. Joint Voxelization for LSTM Fusion:
   • Applies ego-motion compensation for $P_t \in \mathbb{R}^{N\times 3}$2.
   • Merges $P_t \in \mathbb{R}^{N\times 3}$3, $P_t \in \mathbb{R}^{N\times 3}$4, $P_t \in \mathbb{R}^{N\times 3}$5; voxelizes them together.
   • Concatenates backed features within each occupied voxel: $P_t \in \mathbb{R}^{N\times 3}$6.
4. 3D Sparse-Conv LSTM Module:
   • Gate computations use a lightweight U-Net ($P_t \in \mathbb{R}^{N\times 3}$7 encoder block, $P_t \in \mathbb{R}^{N\times 3}$8 bottleneck, $P_t \in \mathbb{R}^{N\times 3}$9 decoder block).
   • Updates memory $N \approx 1.8\times 10^5$0 and hidden state $N \approx 1.8\times 10^5$1 via 3D convolutional LSTM gates.
   • De-voxelizes $N \approx 1.8\times 10^5$2 to point sets of $N \approx 1.8\times 10^5$3.
5. Per-Point 3D Bounding Box Proposal Head:
   • Input: $N \approx 1.8\times 10^5$4.
   • Three $N \approx 1.8\times 10^5$5 sparse conv layers per attribute (center, size, rotation, objectness).
   • De-voxelizes predictions to all points; uses integrated box-corner regression and dynamic objectness classification.
6. Graph-Convolution Smoothing and NMS:
   • Builds a $N \approx 1.8\times 10^5$6-NN graph ($N \approx 1.8\times 10^5$7) among detected centers.
   • Propagates/re-weights box predictions via learned edge weights.
   • Farthest-point sampling yields the top $N \approx 1.8\times 10^5$8 ($N \approx 1.8\times 10^5$9512) proposals.
   • 3D non-maximum suppression with $H_{t-1}$0 produces final detections $H_{t-1}$1.
7. Memory Sub-selection:
   • Filters top $H_{t-1}$2 points from $H_{t-1}$3 by objectness score, yielding $H_{t-1}$4 and $H_{t-1}$5 for the next frame.

2. Sparse-Conv LSTM Formulation

LSTM cell operations are reformulated with sparse 3D U-Net convolutions, substituting standard fully connected gates with spatially aware gated updates: \begin{align*} i_t &= \sigma(W_i \ast [x_t, h_{t-1}] + b_i) \ f_t &= \sigma(W_f \ast [x_t, h_{t-1}] + b_f) \ \tilde{c}t &= \tanh(W_c \ast [x_t, h{t-1}] + b_c) \ c_t &= f_t \odot c_{t-1} + i_t \odot \tilde{c}t \ o_t &= \sigma(W_o \ast [x_t, h{t-1}] + b_o) \ h_t &= o_t \odot \tanh(c_t) \end{align*} Here, “$H_{t-1}$6” denotes 3D sparse convolution, and all gate and cell outputs are spatial feature volumes, not vectors.

Definitions and Typical Dimensions:

Variable	Definition	Dimension
$H_{t-1}$7	Raw LiDAR points	$H_{t-1}$8 ($H_{t-1}$9)
$C_{t-1}$0	Per-point features	$C_{t-1}$1
$C_{t-1}$2	Hidden/cell features	$C_{t-1}$3 ($C_{t-1}$4)
$C_{t-1}$5	Sequence length	typically $C_{t-1}$6 (range $C_{t-1}$7–$C_{t-1}$8)

The spatially structured LSTM is realized by one $C_{t-1}$9-channel encoder, one $N' \approx 3\times 10^4$0-channel bottleneck, one $N' \approx 3\times 10^4$1-channel decoder (U-Net configuration).

3. 3D Proposal Regression, Losses, and Training Objective

Each candidate point regresses:

• Center: $N' \approx 3\times 10^4$2
• Size: $N' \approx 3\times 10^4$3
• Rotation: $N' \approx 3\times 10^4$4 ($N' \approx 3\times 10^4$5 matrix)
• Objectness score: $N' \approx 3\times 10^4$6

Box corners are computed in a differentiable manner for corner-based regression. The total loss function combines regression and classification over all sequence steps: $N' \approx 3\times 10^4$7 where $N' \approx 3\times 10^4$8 are predicted and ground-truth box corners.

The objectness classification loss uses dynamic ground-truth assignment ($N' \approx 3\times 10^4$9 if $P_t$0, $P_t$1 otherwise): $P_t$2

The training objective per frame $P_t$3 is: $P_t$4 with $P_t$5 and $P_t$6, summed across $P_t$7 to $P_t$8 in the temporal window.

4. Empirical Performance and Comparison

Waymo Open Dataset, mAP@0.7:

Method	[email protected] (%)
Single-frame (SparseConv U-Net)	56.1
+ Kalman Filter tracking baseline	56.8
4-frame early-fusion (concatenation)	62.4
4-frame LSTM fusion (Object-Sequenced)	63.6
StarNet (point-based)	53.7
PointPillars	57.2
MVF (multi-view fusion)	62.9

The Object-Sequenced LSTM improves by $P_t$9 mAP over the single-frame method, and $0.2~\text{m}^3$0 over 4-frame early concatenation. Compared to other state-of-the-art approaches in the literature, the LSTM model achieves the highest reported mAP, demonstrating the advantage of learned temporal aggregation in sparse 3D domains (Huang et al., 2020).

Runtime/Memory Efficiency:

• Single-frame backbone: $0.2~\text{m}^3$1 ms/frame (Titan V GPU).
• LSTM:~$0.2~\text{m}^3$2 ms overhead from 3 extra sparse conv blocks.
• Full pipeline: $0.2~\text{m}^3$3100 ms per frame (supports $0.2~\text{m}^3$4 Hz operation).
• Memory: sub-sampled $0.2~\text{m}^3$5 ($0.2~\text{m}^3$6k points), yielding $0.2~\text{m}^3$7k voxels per frame.

5. Implementation Decisions and Ablation Insights

Key hyperparameter and architectural ablation findings:

• Sequence Length ($0.2~\text{m}^3$8):
  • $0.2~\text{m}^3$9 (LSTM, no memory): $3\times 3\times 3$0 mAP.
  • $3\times 3\times 3$1: $3\times 3\times 3$2.
  • $3\times 3\times 3$3 (recommended): $3\times 3\times 3$4.
  • $3\times 3\times 3$5: $3\times 3\times 3$6 (saturation). This suggests moderate temporal horizon is optimal.
• LSTM Module Structure:
  • Replacing fully connected gates with a 3D sparse U-Net in LSTM gates improves performance by $3\times 3\times 3$7 mAP at $3\times 3\times 3$8, highlighting the importance of spatial context within the LSTM gating mechanism.
• Memory Sub-sampling: Sub-sampling $3\times 3\times 3$9 to $[64, 96, 128, 160, 192, 224, 256]$0k points per frame retains object history and maintains constant memory usage, a notable advantage over denser fusion schemes.
• Graph-conv Smoothing: Applies neighborhood-based refinement, reducing jitter and local false positives.
• Early Fusion Concatenation: While denser (more compute/memory), lags LSTM-based fusion by $[64, 96, 128, 160, 192, 224, 256]$1 and does not leverage temporally consistent memory.
• LSTM U-Net Depth: Deeper U-Net variants trade capacity for latency; the $[64, 96, 128, 160, 192, 224, 256]$2-encoder, $[64, 96, 128, 160, 192, 224, 256]$3-bottleneck, $[64, 96, 128, 160, 192, 224, 256]$4-decoder ($[64, 96, 128, 160, 192, 224, 256]$5–$[64, 96, 128, 160, 192, 224, 256]$6 channels) configuration provides a robust capacity-latency balance.

6. Significance in 3D Spatiotemporal Detection

The Object-Sequenced LSTM architecture is the first reported method to apply LSTM mechanisms—augmented with 3D sparse convolutions—for temporal 3D object detection in large LiDAR point clouds. By integrating geometric, sequential, and memory signals within a unified end-to-end pipeline, it sets a new state-of-the-art on major benchmarks while remaining within stringent real-time and computational constraints. The framework’s explicit modeling of temporally evolving objectness mitigates the limitations of both per-frame and naive multi-frame fusion approaches (Huang et al., 2020).

A plausible implication is that this approach generalizes to other sparse sequential 3D domains where spatial and temporal context are both critical.

7. Limitations and Directions for Future Research

Empirical saturation occurs at moderate temporal window lengths ($[64, 96, 128, 160, 192, 224, 256]$7), indicating diminishing returns for longer memory, possibly due to information loss from sub-sampling or temporal drift. While memory sub-sampling is effective, more adaptive or adaptive point selection strategies may further enhance long-range temporal modeling. Exploration of more expressive graph convolution heads or hybrid attention modules presents a potential avenue for improving robustness to local ambiguities.

The demonstrated efficiency and performance of Object-Sequenced LSTM motivate extensions to multi-modal sensor fusion (e.g., radar-LiDAR), domain adaptation, and lifelong temporal learning in robotics and autonomous navigation (Huang et al., 2020).