SemVecMap: Autonomous HD Mapping

• SemVecMap is a modular HD-map generation framework that creates probabilistic BEV semantic maps and vectorized map elements from multi-sensor data.
• Its two-stage architecture first fuses camera and LiDAR inputs via explicit projective geometry and recursive Bayes filtering, then vectorizes features using a MapTRv2-based decoder.
• Deployment studies show that fine-tuning and incremental map updates improve cross-dataset performance, despite minor information loss in the intermediate representation.

Searching arXiv for the cited SemVecMap/SemVecNet papers and closely related context. Calling arXiv search for ([2405.00250](/papers/2405.00250)) SemVecNet and ([2509.25542](/papers/2509.25542)) SemVecMap campus online mapping. SemVecMap denotes, in the autonomous-driving literature, a two-stage online vector map generation pipeline that constructs a probabilistic bird’s-eye-view semantic map from multi-sensor observations and then converts that intermediate representation into vectorized HD-map elements with a MapTRv2-based decoder. The formulation was introduced under the name SemVecNet and explicitly described as “SemVecMap,” with the central goal of improving generalization across arbitrary sensor configurations by avoiding end-to-end overfitting to a fixed camera/LiDAR layout (Ranganatha et al., 2024). A later campus deployment study treats SemVecMap as the onboard mapping model and extends it with fine-tuning and incremental map updates in a real autonomous-driving platform (Zhang et al., 29 Sep 2025). The name also appears in an unrelated 2015 knowledge-representation work, where SemVecMap refers instead to a unified semantic embedding framework built around a “Global Knowledge Map” and an adaptive neuroimaging decoder; that usage is conceptually distinct from the autonomous-driving system (Filatov et al., 2015).

1. Terminology and scope

In the vector-mapping context, SemVecMap is a modular architecture for HD-map generation from synchronized camera images, LiDAR, and an accumulated semantic-map prior. Its defining design choice is the insertion of a geometry-grounded BEV semantic representation between perception and vectorization, rather than learning a direct end-to-end sensor-to-vector transformation (Ranganatha et al., 2024).

This usage should be distinguished from the earlier knowledge-representation formulation. In that separate line of work, SemVecMap is a unified semantic embedding framework whose two components are a self-organizing, high-dimensional “Global Knowledge Map” for documents and concepts, and an adaptive decoder that maps fMRI-derived cognitive patterns into that same vector space (Filatov et al., 2015). The shared name can create a common misconception that all references to SemVecMap concern semantic embeddings or brain–machine interfaces; in current autonomous-driving research, however, the term refers to online vector map generation, sensor generalization, and map maintenance.

A plausible implication of this naming overlap is that citations to “SemVecMap” require domain disambiguation. In autonomous driving, the relevant papers are the SemVecNet introduction and the later campus deployment; in knowledge representation, the relevant reference is the 2015 vector-space mapping proposal.

2. Two-stage architecture and data flow

SemVecMap is organized as a two-stage pipeline. The first stage builds a probabilistic BEV semantic map from multi-view images and LiDAR. The second stage vectorizes that BEV map into map elements such as crosswalks, lane dividers, lane boundaries, and centerlines (Ranganatha et al., 2024).

The original SemVecNet formulation takes as input $n$ synchronized camera images $I_t=\{I_1,\ldots,I_n\}$, a LiDAR point cloud $L_t$ at time $t$, and a semantic-map prior $M_{t-1}$, initialized to zero at $t=0$. The real-time probabilistic BEV semantic mapping module fuses image semantics and LiDAR geometry into an ego-centric bird’s-eye-view grid $S_t$ with class channels $\{\text{road, lane, crosswalk, sidewalk}\}$. This mapping stage comprises three submodules: semantic segmentation, semantic association, and probabilistic mapping. Semantic segmentation uses HRNet+OCR to produce per-pixel labels $G_t$ from each camera image. Semantic association projects each LiDAR point into the image planes and transfers the image-derived semantic label to the point cloud. Probabilistic mapping integrates the labeled point cloud into a BEV grid through a recursive Bayes-filter update per cell.

Once $S_t$ is constructed, it is rasterized into a feature map $I_t=\{I_1,\ldots,I_n\}$0 via a ResNet-50 encoder with reduced strides to preserve resolution, and a transformer-style MapTRv2 decoder converts $I_t=\{I_1,\ldots,I_n\}$1 into vector primitives. The paper summarizes the overall flow as: multi-view images + LiDAR $I_t=\{I_1,\ldots,I_n\}$2 probabilistic semantic BEV map $I_t=\{I_1,\ldots,I_n\}$3 MapTRv2 decoder $I_t=\{I_1,\ldots,I_n\}$4 vector map elements (Ranganatha et al., 2024).

The campus deployment instantiates the same two-stage structure with platform-specific settings. It uses two front-facing RGB cameras $I_t=\{I_1,\ldots,I_n\}$5 at resolution $I_t=\{I_1,\ldots,I_n\}$6 and a synchronized Velodyne VLP-16 LiDAR sweep $I_t=\{I_1,\ldots,I_n\}$7 of approximately $I_t=\{I_1,\ldots,I_n\}$8 K points. Stage 1 employs MScale-HRNet, pretrained on Cityscapes, to produce per-pixel logits for $I_t=\{I_1,\ldots,I_n\}$9 semantic classes, followed by projection and rasterization into a BEV semantic map covering a $L_t$0 m area at $L_t$1 m resolution. Stage 2 applies a ResNet-50 BEV encoder to the one-hot encoding of $L_t$2, then a transformer decoder with instance queries, point queries, and hierarchical queries to produce a set of predicted vectorized polylines $L_t$3 (Zhang et al., 29 Sep 2025).

3. Geometric, probabilistic, and optimization formulation

The semantic association step uses explicit projective geometry. Given a 3D LiDAR point $L_t$4 in homogeneous coordinates, its correspondence in camera $L_t$5 is

$L_t$6

This projection is used to sample a semantic label $L_t$7 from the image-space segmentation output. The labeled point cloud is then accumulated into a BEV grid using a recursive Bayes update. For a cell $L_t$8 and class $L_t$9, with prior $t$0, the posterior is

$t$1

where $t$2 is the observed semantic label, $t$3 is the LiDAR intensity, and $t$4 and $t$5 are pre-computed confusion and intensity priors. After all LiDAR points are integrated, the semantic map is obtained by $t$6 (Ranganatha et al., 2024).

The vectorization stage adopts the hierarchical query design of MapTRv2. For each map element, SemVecMap uses instance-level queries $t$7 and point-level queries $t$8, combined as

$t$9

Self-attention mixes information across instance and point queries, and cross-attention attends from $M_{t-1}$0 onto $M_{t-1}$1 to extract spatial features. The output heads predict an element class label $M_{t-1}$2 and a sequence of 2D BEV coordinates $M_{t-1}$3 for the $M_{t-1}$4-th point of the $M_{t-1}$5-th element (Ranganatha et al., 2024).

SemVecNet trains this stage with Hungarian matching and a composite loss,

$M_{t-1}$6

with $M_{t-1}$7, $M_{t-1}$8, $M_{t-1}$9, and $t=0$0. The deployment paper reports a closely aligned fine-tuning objective consisting of focal-loss instance classification, $t=0$1 vertex regression, and edge-direction consistency, combined as

$t=0$2

with $t=0$3 and $t=0$4; if the semantic stage is jointly fine-tuned, a cross-entropy loss $t=0$5 is added with $t=0$6 (Zhang et al., 29 Sep 2025).

4. Evaluation protocol and cross-dataset behavior

The original evaluation uses NuScenes, Argoverse2, and real-world UCSD campus data. NuScenes comprises $t=0$7 scenes with $t=0$8 cameras and $t=0$9-channel LiDAR, sampled at $S_t$0 Hz. Argoverse2 comprises $S_t$1 train/val/test logs with $S_t$2 cameras and $S_t$3-channel LiDARs, sampled at $S_t$4 Hz. The metric is mean Average Precision over four classes—ped cross, lane divider, boundary, and centerline—at Chamfer-distance thresholds $S_t$5, with cross-dataset transfer summarized through the ratio $S_t$6 when training on dataset $S_t$7 and evaluating on dataset $S_t$8 (Ranganatha et al., 2024).

Train $S_t$9 Test	SemVecNet mAP	MapTRv2 mAP
AV2 $\{\text{road, lane, crosswalk, sidewalk}\}$0 AV2	49.0	67.4
AV2 $\{\text{road, lane, crosswalk, sidewalk}\}$1 NuSc	12.2	0
NuSc $\{\text{road, lane, crosswalk, sidewalk}\}$2 NuSc	48.8	61.5
NuSc $\{\text{road, lane, crosswalk, sidewalk}\}$3 AV2	16.2	0

These results establish the central empirical claim of SemVecMap: its cross-dataset transfer is non-zero, whereas the fully end-to-end MapTRv2 baseline drops to $\{\text{road, lane, crosswalk, sidewalk}\}$4 under the reported train/test swaps. The corresponding transfer ratios are $\{\text{road, lane, crosswalk, sidewalk}\}$5 for AV2 $\{\text{road, lane, crosswalk, sidewalk}\}$6 NuSc and $\{\text{road, lane, crosswalk, sidewalk}\}$7 for NuSc $\{\text{road, lane, crosswalk, sidewalk}\}$8 AV2. At the same time, the in-domain numbers show that SemVecNet remains below MapTRv2 when the sensor configuration is unchanged, which the paper explicitly attributes to possible information loss in the intermediate representation.

The ablation studies on Argoverse2 further localize the behavior of the pipeline. Using a single front camera yields mAP $\{\text{road, lane, crosswalk, sidewalk}\}$9, whereas a 6-camera surround configuration yields mAP $G_t$0, reported as a $G_t$1 drop when only one camera is used. For BEV grid resolution, a $G_t$2 m grid gives mAP $G_t$3 and a $G_t$4 m grid gives mAP $G_t$5, indicating no significant gain at higher resolution. In the UCSD campus evaluation, without any fine-tuning, SemVecMap produces qualitatively correct centerlines and boundaries, which the paper presents as evidence of robustness to novel sensor layouts (Ranganatha et al., 2024).

5. Campus deployment, fine-tuning, and dynamic map updates

The campus deployment places SemVecMap on a GEM e6 vehicle with drive-by-wire, two NVIDIA RTX3070 GPUs, and ROS-based middleware. Camera and LiDAR sensor nodes publish at $G_t$6 Hz, and an approximate_time synchronizer downsamples to $G_t$7 Hz for mapping. Per time step, MScale-HRNet runs on GPU 1 in approximately $G_t$8 ms, semantic point-cloud formation and rasterization take approximately $G_t$9 ms on CPU, and ResNet50 plus the transformer decoder run on GPU 2 in approximately $S_t$0 ms, for an end-to-end latency of approximately $S_t$1 ms. The system publishes vector elements both to the Autoware planner and to a map-update module, and the paper states that this supports $S_t$2–$S_t$3 Hz full-pipeline operation if desired (Zhang et al., 29 Sep 2025).

Fine-tuning uses four campus regions—straight roads, intersections, loops, roundabouts, and multi-lane roads—over a total one-way track of approximately $S_t$4 km. The dataset contains $S_t$5 train and $S_t$6 test scenes after downsampling raw $S_t$7 Hz logs to $S_t$8 Hz and applying a $S_t$9 split per scenario. Augmentation includes random horizontal flip at $I_t=\{I_1,\ldots,I_n\}$00, brightness/contrast jitter of $I_t=\{I_1,\ldots,I_n\}$01, BEV rotation of $I_t=\{I_1,\ldots,I_n\}$02, scale in $I_t=\{I_1,\ldots,I_n\}$03, and LiDAR Gaussian noise with $I_t=\{I_1,\ldots,I_n\}$04 m. Domain adaptation mixes in $I_t=\{I_1,\ldots,I_n\}$05 NuScenes frames during mini-batch sampling while oversampling campus frames to maintain a $I_t=\{I_1,\ldots,I_n\}$06 campus / $I_t=\{I_1,\ldots,I_n\}$07 nuScenes balance. Optimization uses AdamW with initial learning rate $I_t=\{I_1,\ldots,I_n\}$08, $I_t=\{I_1,\ldots,I_n\}$09, $I_t=\{I_1,\ldots,I_n\}$10, weight decay $I_t=\{I_1,\ldots,I_n\}$11, batch size $I_t=\{I_1,\ldots,I_n\}$12, $I_t=\{I_1,\ldots,I_n\}$13 epochs, and a learning-rate decay by $I_t=\{I_1,\ldots,I_n\}$14 at epoch $I_t=\{I_1,\ldots,I_n\}$15 (Zhang et al., 29 Sep 2025).

The incremental map-update mechanism maintains an accumulation grid $I_t=\{I_1,\ldots,I_n\}$16 at $I_t=\{I_1,\ldots,I_n\}$17 m resolution. For each incoming frame and predicted line, the system transforms ego-frame points to world coordinates, increments the corresponding grid cells, thresholds $I_t=\{I_1,\ldots,I_n\}$18 to form a mask, skeletonizes the mask into a set of new polylines $I_t=\{I_1,\ldots,I_n\}$19, and then partitions the world into $I_t=\{I_1,\ldots,I_n\}$20 m tiles. For each tile $I_t=\{I_1,\ldots,I_n\}$21, it computes mean Average Precision between $I_t=\{I_1,\ldots,I_n\}$22 and the existing HD map $I_t=\{I_1,\ldots,I_n\}$23 at Chamfer thresholds $I_t=\{I_1,\ldots,I_n\}$24. If $I_t=\{I_1,\ldots,I_n\}$25, with an example $I_t=\{I_1,\ldots,I_n\}$26, the tile is marked for update, and $I_t=\{I_1,\ldots,I_n\}$27 is overlaid onto $I_t=\{I_1,\ldots,I_n\}$28 using Autoware Tools with endpoint snapping $I_t=\{I_1,\ldots,I_n\}$29 m and conflict resolution favoring the newest geometry (Zhang et al., 29 Sep 2025).

Quantitatively, SemVecNet trained only on NuScenes reaches campus-test mAP $I_t=\{I_1,\ldots,I_n\}$30, with $I_t=\{I_1,\ldots,I_n\}$31 boundary, $I_t=\{I_1,\ldots,I_n\}$32 divider, and $I_t=\{I_1,\ldots,I_n\}$33 crosswalk. After adding campus data in training, the campus-test mAP rises to $I_t=\{I_1,\ldots,I_n\}$34, with $I_t=\{I_1,\ldots,I_n\}$35, $I_t=\{I_1,\ldots,I_n\}$36, and $I_t=\{I_1,\ldots,I_n\}$37 for those classes. The paper also reports per-scenario gains:

Scenario	Pre	Post
Straight roads	12.08	60.08
Intersections	14.40	56.09
Loops	5.82	47.51
Roundabouts	5.02	41.40
Multi-lane	14.87	87.89

In a reported HD-map update case, fused heatmaps from $I_t=\{I_1,\ldots,I_n\}$38 frames in a previously unseen construction zone produce a final polyline map with Chamfer-based IoU of approximately $I_t=\{I_1,\ldots,I_n\}$39 against manually refined ground truth, while capturing temporary lane closures and barrier lines (Zhang et al., 29 Sep 2025).

6. Generalization rationale, limitations, and research context

The core rationale for SemVecMap is that a BEV semantic intermediate improves sensor generalization because it avoids end-to-end learned “view-transform” overfitting to a single camera/LiDAR layout, uses explicit 3D-geometry-based projection via known intrinsics and extrinsics, and benefits from temporal accumulation of LiDAR and semantic labels to fill occlusions and obtain dense BEV coverage under changing viewpoints (Ranganatha et al., 2024).

This design does not remove all domain gaps. The reported limitations are specific and technically consequential. LiDAR density shifts, such as Argoverse2’s two LiDARs versus NuScenes’s one, cause “holes” in semantic grids. Road-structure distribution differences, including U.S. versus Singapore crosswalk conventions in NuScenes, create class-specific biases such as over-prediction of crosswalks. The intermediate representation also lowers in-domain performance relative to fully end-to-end MapTRv2, suggesting information loss between perception and vectorization. The stated future direction is to replace discrete semantic grids with neural or probabilistic feature fields in order to reduce the information bottleneck (Ranganatha et al., 2024).

These limitations delimit a common misunderstanding. SemVecMap is not presented as universally sensor-agnostic in the sense of eliminating retraining or dataset shift; rather, it is presented as a modular pipeline that generalizes significantly better than end-to-end baselines under unseen sensor configurations, while still exhibiting measurable gaps under LiDAR-density changes, road-distribution shifts, and same-domain accuracy comparisons.

In the broader research landscape represented by the cited papers, SemVecMap occupies a specific position: it is a geometry-grounded online HD-mapping framework for autonomous driving, not a generic semantic vector-space method. The separate 2015 knowledge-representation usage of the name concerns unsupervised corpus embeddings, Johnson–Lindenstrauss-style dimensionality reduction, and an adaptive decoder from fMRI patterns to a “Global Knowledge Map” (Filatov et al., 2015). The coexistence of these two meanings makes terminological precision necessary, but it does not alter the technical identity of SemVecMap in contemporary autonomous-driving work: a two-stage system that couples probabilistic semantic BEV mapping with transformer-based vectorization and, in deployment, supports continual HD-map updates (Zhang et al., 29 Sep 2025).