DenseTNT: Dense Prediction & Classification

• The paper introduces a dense, anchor-free trajectory prediction model that leverages a fine-grained goal grid and probabilistic decoding to improve minADE and miss rate metrics.
• DenseTNT integrates deep learning modules, using GNNs, transformers, and pseudo-labeling to capture both local details and global context for enhanced forecasting and vehicle classification.
• Empirical evaluations demonstrate state-of-the-art performance on benchmarks like Argoverse and Waymo and robust accuracy under challenging conditions such as fog and off-center trajectories.

DenseTNT refers to two distinct but high-impact models: (1) an anchor-free, dense goal-based multi-trajectory prediction architecture for autonomous driving, and (2) a Densely Connected Convolutional Transformer-in-Transformer Neural Network for robust vehicle type classification from satellite imagery. Both models represent advances in leveraging dense representations combined with deep learning modules—attention in road agent forecasting and hierarchical transformers in remote vehicle perception.

1. DenseTNT for Trajectory Prediction in Autonomous Driving

Problem Formulation and Motivation

DenseTNT, or "Dense Target-driven Trajectory Prediction," addresses the problem of multimodal trajectory forecasting for autonomous vehicles. Given an observed trajectory $x_{1:T_0}$ of a focal road agent in bird’s-eye view (BEV), the task is to predict $K$ plausible future trajectories $\{\hat{y}_{1:H}^k\}_{k=1}^K$ over a horizon $H$. Trajectory prediction is modeled as a two-step process: $$P(\text{trajectory}) \approx P(\text{goal}=c_i) \cdot P(\text{trajectory}\mid \text{goal}=c_i)$$ where $\{c_i\}_{i=1}^N$ is a dense set of candidate goal positions sampled from drivable areas on the HD map (Gu et al., 2021, Gu et al., 2021).

Traditional goal-based predictors (e.g., TNT) use sparse, heuristic anchors. DenseTNT replaces these with a dense, anchor-free grid over the drivable surface, addressing two key limitations:

• Sparse anchors limit multimodal expressivity to pre-specified modes.
• Off-center or non-centerline feasible endpoints are unreachable in sparse anchor formats.

2. Dense Goal Set Construction and Scene Encoding

The dense candidate goal set $C$ is constructed by:

• Filtering lanes using a binary classifier (supervised by binary cross-entropy), keeping only lanes with high likelihood of containing the true goal.
• Sampling points at regular spatial intervals (e.g., $\delta=1$ m) along each selected lane, resulting in fine-grained $C = \{c_i\}_{i=1}^N$ (Gu et al., 2021).

DenseTNT employs a scene context encoder based on VectorNet-style GNNs—each lane or agent polyline is processed through a subgraph and then aggregated globally. This provides rich relational features $L \in \mathbb{R}^{M \times d_h}$ for subsequent stages (Gu et al., 2021).

3. Probabilistic Modeling of Dense Goals and Trajectory Decoding

Each candidate $c_i$ is embedded via an MLP, and attends to scene context: $$A = \text{softmax}\left(\frac{Q K^T}{\sqrt{d_k}}\right) V$$ with $Q=F W^Q$ and $K,V$ as context projections. Scores $\alpha_i = g([F_i ; A_i])$ are normalized by softmax to yield a probability heatmap $\phi_i$ over goals: $$\phi_i = \frac{\exp(\alpha_i)}{\sum_{n=1}^N \exp(\alpha_n)}$$ (Gu et al., 2021, Gu et al., 2021). Binary cross-entropy loss enforces the goal probability peak near ground-truth endpoints.

Goal selection in DenseTNT is performed via either:

• Non-maximum suppression (NMS) over $\phi_i$ (baseline instantiation).
• A learned set-prediction head, using $H'$ parallel “heads” for end-to-end $K$-goal prediction, where each head encodes the goal heatmap and outputs coordinate/confidence tuples; the most confident head is selected for inference (Gu et al., 2021).

For each selected goal, a trajectory decoder reconstructs future positions as offsets from the endpoint: $\hat{y}_t = y_H + \Delta \hat{y}_t$ using a dedicated trajectory decoder MLP, with smooth L1 loss against the full ground truth for training.

4. Multi-Future Pseudo-Labeling and Training Procedures

DenseTNT introduces an offline optimization-based pseudo-labeling technique to address the challenge of single-future observation at train time versus the need to supervise multiple ($K$) predictions. The method searches for a set of $K$ goals that minimizes expected set distance, using hill-climbing and random perturbations within a given time budget (e.g., 100 ms), with final sets taken as pseudo-labels for supervising the online model (Gu et al., 2021).

Training proceeds in two stages:

• Stage 1: Jointly optimizes the context encoder, goal encoder, and trajectory decoder with lane, goal, and trajectory completion losses.
• Stage 2: Trains the goal-set predictor using offline pseudo-labels, supervising only the head yielding lowest expected error, and using a head-confidence binary cross-entropy (Gu et al., 2021).

5. Empirical Performance and Ablation Studies

DenseTNT demonstrates state-of-the-art results on public motion forecasting challenges:

• Argoverse (K=6): DenseTNT (online) achieves minADE 0.82, minFDE 1.37, MR 7.0%; the offline optimizer improves minFDE to 1.27 and MR to 7.0%. The best leaderboard MR is 10.7% (1st place) (Gu et al., 2021).
• Waymo Open Dataset: mAP=0.3281 (1st place); minADE=1.0387 m, minFDE=1.5514 m, MissRate 0.1573 (Gu et al., 2021).

Ablation studies confirm that denser sampling (from 3 m to 1 m) and offline optimization decrease minFDE and MR significantly. The two-stage set-predictor removes the need for any heuristic NMS post-processing (Gu et al., 2021).

Model Variant	minADE	minFDE	MissRate
TNT (Sparse+NMS)	0.82	1.35	9.5%
DenseTNT (online)	0.82	1.37	7.0%
DenseTNT (offline opt)	0.80	1.27	7.0%

6. Dense-TNT for Robust Vehicle Classification from Remote Sensing

Dense-TNT also refers to an efficient vehicle type classification network that fuses a DenseNet backbone (for local detail) with Transformer-in-Transformer (TNT) layers (for global context) (Luo et al., 2022). The architecture processes 64×64 grayscale satellite patches—DenseNet extracts fine features, while TNT applies inner- and outer-transformers to sub-patch and patch-level embeddings, propagating both local and global spatial context.

All layers and update formulas follow those in Han et al.’s TNT, with concatenation and attention operations precisely characterized. The classifier predicts between sedan and pickup.

Empirical evaluation on COWC regions shows that Dense-TNT s24 achieves:

• Accuracy: 0.8065 (Selwyn), 0.7685 (Columbus), 0.8009 (Toronto)
• F1-score: 0.8810 (Selwyn), 0.8582 (Columbus), 0.8734 (Toronto)

Under heavy fog simulation, Dense-TNT’s accuracy drop is ≲4%, compared to 7–12% for other deep vision baselines (Luo et al., 2022).

7. Limitations and Prospective Developments

DenseTNT’s trajectory prediction framework:

• Is limited to endpoint sampling on drivable area geometry, with generalization constrained by sampling density and lane-filtering accuracy.
• Relies on discrete pseudo-label optimization; continuous endpoint learning could further improve coverage.
• Evaluated primarily on vehicle agents, though the architecture generalizes to pedestrians and cyclists (separately trained) (Gu et al., 2021, Gu et al., 2021).

In the vehicle classification variant, Dense-TNT is currently binary (sedan/pickup), simulates grayscale-only fog, and its parameter count (12–21 M) may exceed nano-satellite on-board limits. Future directions include multi-class extension, incorporation of learnable dehazing, pruning for edge deployment, and fusion with multi-spectral imagery (Luo et al., 2022).