---
title: 'HDMapNet: Online HD Semantic Mapping'
url: https://www.emergentmind.com/topics/hdmapnet
type: topic
---

# HDMapNet: Online HD Semantic Mapping

HDMapNet is an online semantic map learning framework designed to dynamically construct high-definition (HD) vectorized road semantics from onboard sensor measurements in autonomous driving scenarios. Unlike traditional HD mapping pipelines reliant on extensive manual annotation and resource-intensive survey processes, HDMapNet predicts lane boundaries, dividers, and pedestrian crossings directly from multi-modal inputs, enabling scalable map generation in the vehicle’s local vicinity.

## 1. Formulation of HD Semantic Map Learning

HD semantic map learning targets the prediction of vectorized map elements $M$ (lane boundaries, dividers, crosswalks) in the bird’s-eye view (BEV) frame from sensor observations consisting of multi-camera images $I$ and/or LiDAR point clouds $P$. The mapping function is:

$$
\hat{M} = \text{HDMapNet}(I, P)
$$

HDMapNet decomposes into four main neural modules:

- $\varphi_I$: Perspective-view image encoder
- $\varphi_V$: Neural view transformer (perspective/camera $\rightarrow$ BEV)
- $\varphi_P$: Pillar-based LiDAR encoder
- $\varphi_M$: BEV decoder yielding semantic segmentation, instance embeddings, and direction fields

Supervised training employs losses on BEV semantics, instance clusters, and direction labels.

## 2. Network Architecture and Modalities

HDMapNet utilizes a multi-branch input structure supporting camera-only, LiDAR-only, or camera-LiDAR fusion modalities, all converging into a unified BEV decoder. The architecture encompasses:

- **Camera branch**: $N_m$ surround-camera images processed via EfficientNet-B0 (pre-trained on ImageNet) for perspective-view features $F_I^{pv}$.
- **View transformation**: For each camera, an MLP $\varphi_{V_i}$ projects perspective-grid locations to small camera-coordinate grids, then warps features into ego-vehicle BEV; averaged across $N_m$ cameras for $F_I^{bev}$.
- **LiDAR branch**: 32-beam LiDAR input, dynamically voxelized into pillars $P_j$, each processed with PointNet aggregation to yield pillar features $f_{\text{pillar}_j}$ and encoded via a dedicated 2D CNN for BEV feature maps $F_P^{bev}$.
- **BEV decoder**: Fully convolutional (ResNet-style), with three simultaneous output heads—semantic segmentation (per-pixel softmax), instance embedding ($\mathbb{R}^E$), and direction classification (discretized into $N_d$ bins).

Intermediate resolutions include $H_{pv}\times W_{pv}$ for perspective views (typically $64\times256$) and $H_{bev}\times W_{bev}$ for BEV grids (typically $200\times200$).

## 3. Training Objectives

HDMapNet employs a composite training loss comprising:

- **Semantic segmentation loss (cross-entropy):**
  $$
  L_{\text{sem}} = -\sum_{x\in\Omega}\sum_{c=1}^C y_c(x)\log p_c(x)
  $$
- **Discriminative instance embedding loss** [De Brabandere et al., 2017], with cluster variance and separation terms and margin parameters $\delta_v$, $\delta_d$:
  $$
  L_{\text{var}} = \frac{1}{C} \sum_{c=1}^C \frac{1}{N_c}\sum_{j=1}^{N_c}\left[\|f_j^{\text{inst}}-\mu_c\|-\delta_v\right]_+^2
  $$
  $$
  L_{\text{dist}} = \frac{1}{C(C-1)}\sum_{c_A\neq c_B}[2\delta_d-\|\mu_{c_A}-\mu_{c_B}\|]_+^2
  $$
  $$
  L_{\text{inst}} = \alpha L_{\text{var}} + \beta L_{\text{dist}}
  $$
- **Direction classification loss:** Per-pixel softmax cross-entropy on $N_d$ direction bins, applied only at lane pixels.

Full training objective:
$$
L = L_{\text{sem}} + L_{\text{inst}} + L_{\text{dir}}
$$
Recommended hyperparameters: $\alpha=1$, $\beta=1$, $\delta_v=0.5$, $\delta_d=3.0$.

## 4. Output Map Representation and Vectorization Workflow

The BEV decoder produces three grids for each pixel $x$:

- $S(x)$: Semantic class probability
- $E(x)\in \mathbb{R}^{E}$: Instance embedding
- $D(x)\in \Delta^{N_d}$: Direction logits

Inference post-processing includes:

1. Thresholding $S(x)$ to select lane-mask pixels.
2. Clustering per-pixel embeddings $E(x)$ via DBSCAN to define instance sets $\{S_c\}$.
3. Applying non-maximum suppression to instance confidences.
4. For each instance, tracing the polyline by seed-pixel selection and recursive following of predicted direction $D(x)$.
5. Simplification of polylines (e.g., Ramer–Douglas–Peucker) into ordered vectors of 2D meter-space coordinates.

## 5. Evaluation Protocols and Metrics

### Semantic-Level Metrics

- **Intersection-over-Union (IoU)** over BEV grid:
  $$
  \text{IoU}(D_1, D_2) = \frac{|D_1 \cap D_2|}{|D_1 \cup D_2|}
  $$
- **Chamfer Distance (CD)** for predicted/ground-truth curve sets $S_1$, $S_2$:
  $$
  \text{CD}_{\text{dir}}(S_1, S_2) = \frac{1}{|S_1|}\sum_{x\in S_1}\min_{y\in S_2}\|x-y\|_2
  $$
  $$
  \text{CD}(S_1, S_2) = \text{CD}_{\text{dir}}(S_1, S_2) + \text{CD}_{\text{dir}}(S_2, S_1)
  $$

### Instance-Level Metrics

Instance polylines are considered “objects” for detection evaluation. Average Precision (AP) is computed at variable Chamfer distance thresholds $\tau$, with mAP computed as the mean of AP at ten recall values between $0.1$ and $1.0$. Typical reporting: AP@$0.2$m, AP@$0.5$m, AP@$1.0$m, mean mAP.

## 6. Experimental Setup and Results

### Dataset

- **nuScenes** [Caesar et al., 2020]: 1000 urban driving scenes, each annotated for lane boundaries, dividers, pedestrian crossings in $30\,\mathrm{m}\times 30\,\mathrm{m}$ BEV crops around keyframes. Inputs: 6 surround cameras, 32-beam LiDAR. Splits correspond to standard nuScenes protocol.

### Implementation

- Image branch: EfficientNet-B0 ($K=128$ channels)
- LiDAR branch: PointPillars ($64$-dim pillar features), followed by a pillar CNN to upsample/concatenate to $K=128$
- BEV decoder: ResNet-style FCN with three parallel heads
- Training: Adam optimizer, learning rate $1\mathrm{e}{-3}$, batch size $16$, $30$ epochs, $\times 0.1$ lr decay every $10$ epochs
- BEV grid: $200\times200$ at $0.3\,\mathrm{m}$/pixel ($60\,\mathrm{m}\times 60\,\mathrm{m}$)

### Quantitative Results

| Variant                | Divider IoU | Ped-Crossing IoU | Boundary IoU | All-Classes IoU | mAP ({0.2,0.5,1.0}m) |
|------------------------|-------------|------------------|--------------|-----------------|----------------------|
| HDMapNet(Surr)         | 40.6%       | 18.7%            | 39.5%        | 32.9%           | 22.7%                |
| VPN                    | 36.5%       | 15.8%            | 35.6%        | 29.3%           | 17.5%                |
| Lift-Splat             | 38.3%       | 14.9%            | 39.3%        | 30.8%           | 17.4%                |
| IPM (CB)               | 38.6%       | 19.3%            | 39.3%        | 32.4%           | 19.7%                |
| HDMapNet(Fusion)       | 46.1%       | 31.4%            | 56.0%        | 44.5%           | 30.6%                |

Fusion modality yields approximately $50\%$ relative gain over camera-only variant in semantic IoU and up to $55\%$ absolute mAP improvement over IPM.

**Qualitative observations:** HDMapNet produces visually clean, distortion-free BEV vector maps, robust to nighttime/rain and temporally consistent over $100\,$m local regions.

## 7. Modalities, Advantages, Limitations, and Implications

**Modality complementarity:** Cameras provide enhanced discrimination of color-based semantics (dividers, crosswalks), while LiDAR excels in detecting geometric lane boundaries. Fusion synergistically captures the best features of each.

**Advantages:**
- Enables purely sensor-driven, online local mapping without reliance on global SLAM or manual annotation
- Direct output of vectorized elements is immediately utilizable in motion planning systems

**Limitations and outlook:**
- Absolute accuracy lags behind hand-annotated HD maps, though the scalability trade-off is advantageous
- Current fusion employs naive feature concatenation; more sophisticated strategies (attention mechanisms, gating, etc.) may yield further improvements
- Semantic coverage is restricted to three classes; generalization to additional road infrastructure (traffic signs, poles, curbs) is pending
- Highly dynamic urban environments pose challenges and motivate continual-learning approaches

A plausible implication is that HDMapNet serves as a foundation for scalable, sensor-driven semantic mapping, facilitating map generation without manual intervention and with extensible architecture for future expansion of both input modalities and map element classes [2107.06307].

Source: https://www.emergentmind.com/topics/hdmapnet