VectorNet: Hierarchical GNN for Autonomous Driving

• VectorNet specializes in modeling multi-agent environments in autonomous driving through a hierarchical graph neural network (GNN) architecture.
• It processes vectorized road maps and agent trajectories, overcoming limitations of raster-based ConvNets, with a two-stage graph model for local and global interactions.
• The model reduces parameter count and computational requirements significantly, while outperforming existing approaches in prediction accuracy.

VectorNet is a hierarchical graph neural network (GNN) architecture designed for behavior prediction in dynamic, multi-agent environments, with particular application to autonomous driving systems. It addresses the limitations of raster-based convolutional neural network (ConvNet) approaches by directly operating on vectorized representations of high-definition (HD) maps and agent trajectories. This enables both efficient encoding and precise modeling of spatial relationships and interactions among road elements and moving agents (Gao et al., 2020).

1. Vectorized Representation of Road Context and Agent Dynamics

VectorNet encodes the static and dynamic context of traffic scenes by representing all road components (lane-centers, lane-boundaries, crosswalk polygons, traffic-light points) and agent trajectories as sequences of short vectors—line segments sampled at fixed spatial (for map entities) or temporal (for trajectories) intervals. Each such vector is assigned to a polyline group, producing an unordered set of polyline segments.

The formal representation of each input vector $v_i$ in polyline $\mathcal P_j$ is:

$v_i = [d_i^s,\;d_i^e,\;a_i,\;j]$

where $d_i^s, d_i^e \in \mathbb R^2$ (or $\mathbb R^3$) are the start and end coordinates, $a_i$ encodes semantic and context attributes (e.g., lane type, speed limit, traffic-light state, temporal index), and $j$ is the polyline identifier for subgraph assignment.

Agent-centric normalization translates (and optionally rotates) all coordinates so that the last observed position of the target agent is at the origin. This imposes translational invariance on the feature space.

2. Hierarchical GNN Model Architecture

VectorNet adopts a two-stage hierarchical GNN structure:

2.1 Local Polyline Subgraphs

Each polyline $\mathcal P = \{v_1, \ldots, v_P\}$ is modeled as a fully-connected local subgraph among its constituent vectors. The node update for each vector at layer $l$ is:

$$v_i^{(l+1)} = \varphi_\text{rel}\bigl(g_\text{enc}(v_i^{(l)}),\; \varphi_\text{agg}(\{g_\text{enc}(v_j^{(l)})\}_{j\neq i})\bigr)$$

where $\mathcal P_j$0 is an MLP followed by LayerNorm and ReLU; $\mathcal P_j$1 performs element-wise max-pooling across neighbors; and $\mathcal P_j$2 denotes concatenation. After $\mathcal P_j$3 such layers, a single aggregated polyline embedding $\mathcal P_j$4 (again, max-pooling) is produced.

2.2 Global Polyline Interaction Graph

Polyline embeddings from the local stage act as nodes in a fully-connected global interaction graph. The global update uses self-attention:

$\mathcal P_j$5

where $\mathcal P_j$6 collects all node features, and $\mathcal P_j$7, $\mathcal P_j$8, $\mathcal P_j$9 are linear projections to query, key, and value spaces, respectively. The model uses a single global attention layer ($v_i = [d_i^s,\;d_i^e,\;a_i,\;j]$0).

Decoding of agent future trajectories is performed via a shallow MLP $v_i = [d_i^s,\;d_i^e,\;a_i,\;j]$1 applied to the polyline’s final global feature, outputting per-step offset coordinates.

3. Auxiliary Node Recovery and Training Objective

To enhance context modeling beyond supervised trajectory prediction, VectorNet employs an auxiliary self-supervised node recovery task inspired by masked modeling paradigms.

A random subset of polyline nodes (from both maps and agents) has its input embedding zeroed during training. Each masked node receives an auxiliary “ID” embedding (such as the minimum of its vectors’ start coordinates) to identify it within the graph. The goal is to reconstruct the masked node’s embedding after the global interaction stage:

$v_i = [d_i^s,\;d_i^e,\;a_i,\;j]$2

The corresponding loss is a Huber penalty:

$v_i = [d_i^s,\;d_i^e,\;a_i,\;j]$3

where $v_i = [d_i^s,\;d_i^e,\;a_i,\;j]$4 is the set of masked polylines. All polyline embeddings are $v_i = [d_i^s,\;d_i^e,\;a_i,\;j]$5-normalized before the global GNN layer to prevent trivial solutions.

The primary supervised loss is negative log-likelihood under a per-step Gaussian distribution for future positions:

$v_i = [d_i^s,\;d_i^e,\;a_i,\;j]$6

The total objective is:

$v_i = [d_i^s,\;d_i^e,\;a_i,\;j]$7

with $v_i = [d_i^s,\;d_i^e,\;a_i,\;j]$8.

4. Experimental Protocols and Datasets

The model was evaluated on two benchmarks:

• In-house benchmark: 2.2 million training and 0.55 million test trajectories, with a 1 s observation and 3 s prediction window. Map features included lane-centers, stop signs, crosswalks, and speed bumps.
• Argoverse Forecasting Dataset: 211,000 train, 41,000 validation, and 80,000 test sequences. Observation window of 2 s, prediction window of 3 s, sampled at 10 Hz. The map contains a rich lane graph and driveable area information.

All entities are vectorized per the described protocol, and coordinates are normalized with respect to the target agent.

Training used $v_i = [d_i^s,\;d_i^e,\;a_i,\;j]$9 layers in local subgraphs, $d_i^s, d_i^e \in \mathbb R^2$0 attention layer globally, with hidden dimension 64 throughout. Models were trained for 25 epochs using Adam (initial learning rate 0.001, decaying by 0.3 every 5 epochs) with synchronised batches across 8 GPUs.

5. Quantitative Results and Computational Characteristics

Evaluation metrics include:

• ADE: Average per-step displacement error over the forecast horizon.
• DE@t: Displacement error at time $d_i^s, d_i^e \in \mathbb R^2$1 seconds ($d_i^s, d_i^e \in \mathbb R^2$2), both reported in meters.

VectorNet was compared against a ResNet-18 ConvNet baseline operating on rasterized bird’s-eye representations (400 × 400 pixels, 3N channels: N=10 for in-house, N=20 for Argoverse). A summary of the results:

Dataset	Model	DE@3s (m)	ADE (m)	Parameters	FLOPs per agent
In-house	ConvNet (best)	≈1.00	≈0.63	246K	10.6 GFLOPs
In-house	VectorNet	1.00	0.66	72K	0.041G × n_agents
Argoverse	ConvNet (best)	≈4.49	≈1.96	246K	10.6 GFLOPs
Argoverse	VectorNet	3.67	1.66	72K	0.041G × n_agents

On the Argoverse public leaderboard with $d_i^s, d_i^e \in \mathbb R^2$3 (single trajectory), VectorNet achieved DE@3s = 4.01 m and ADE = 1.81 m, outperforming the previous state of the art (DE@3s ≈ 4.17 m).

VectorNet reduces parameter count by over 70% and the number of floating-point operations (FLOPs) per agent by one to two orders of magnitude compared to the ConvNet baseline.

6. Ablation Studies and Architectural Insights

Ablation analyses revealed several findings:

• Including map and agent context consistently improved performance (in-house DE@3s: no context ≈1.29 m; map alone 1.11 m; map + agents 1.05 m; map + agents + node auxiliary 1.00 m).
• The optimal configuration was $d_i^s, d_i^e \in \mathbb R^2$4 local GNN layers and $d_i^s, d_i^e \in \mathbb R^2$5 global layer. Increasing width (more MLP units) degraded generalization on Argoverse.
• The auxiliary node recovery loss especially improved long-horizon trajectory errors, consistent with bidirectional context modeling seen in LLMs.
• The agent-centric normalization is single-target; multi-target parallel prediction was not explored. Output trajectories are unimodal; variants could introduce anchor-based or variational decoders (e.g., MultiPath, VRNNs). The current fully-connected global graph could become computationally demanding if the number of polylines increases; sparse connectivity heuristics are a potential extension.

7. Significance and Limitations

VectorNet establishes an end-to-end vectorized modeling pipeline that alleviates the computational and representational inefficiencies of raster-based ConvNet approaches. By hierarchically exploiting spatial structure at the subgraph (polyline) and global (roadscape + agent) levels and supporting auxiliary self-supervised learning, it achieves state-of-the-art or competitive predictive accuracy with substantially fewer computational resources.

Current design choices—single target normalization, unimodal output, fully-connected graphs—suggest further research directions in multi-target prediction, multi-modality, and sparse interaction modeling for large-scale traffic scenes (Gao et al., 2020).