Token-Guided 2D Patch Distillation (TGPD)

• The paper introduces a targeted distillation technique that aligns multimodal BEV features through patch tokenization and attention mechanisms.
• TGPD decomposes dense BEV maps into local patch tokens and a global token, using Transformer-style self-attention for focused cross-modal supervision.
• The method significantly accelerates student convergence and improves mIoU and mAP in online HD map construction scenarios.

Token-Guided 2D Patch Distillation (TGPD) is a targeted knowledge distillation strategy for bird’s eye view (BEV) feature alignment introduced in MapKD for efficient online HD map construction [2508.15653]. Within MapKD’s Teacher-Coach-Student (TCS) paradigm, a camera-LiDAR fusion model with SD/HD map priors serves as the teacher, a vision-centric coach model with prior knowledge and simulated LiDAR bridges the cross-modal knowledge transfer gap, and a lightweight vision-based student is optimized for low-cost inference. TGPD addresses the mismatch between the teacher’s multimodal, prior-enriched BEV representations and the student’s camera-only BEV representations by decomposing BEV feature maps into local patch tokens and a global token, applying Transformer-style self-attention to these tokens, and supervising the student through a KL-based alignment of token-attention distributions together with a light MSE penalty on patch features [2508.15653].

1. Role within cross-modal BEV distillation

TGPD is motivated by a specific difficulty in cross-modal knowledge distillation for BEV-map construction: the teacher’s BEV features, fused from LiDAR, camera, and map priors, and the student’s BEV features, derived from cameras only, live in quite different representation spaces [2508.15653]. The method is designed as a response to two limitations stated explicitly in the MapKD description. First, vanilla feature-level distillation, such as MSE on whole feature maps, often forces a low-capacity student to regress dense LiDAR-rich patterns it cannot synthesize from images, leading to slow convergence or suboptimal minima. Second, response-level distillation, such as matching logits, cannot fully convey the rich spatial structures embedded in intermediate BEV representations [2508.15653].

A common misconception in this setting is that denser supervision is necessarily better. TGPD is framed against that assumption. Rather than flooding the student with unnecessary signal over the full BEV tensor, it reduces the regression burden to compact local subregions and shifts emphasis toward where the teacher focuses. Another misconception is that logit matching alone is sufficient for transferring BEV structure. The formulation of TGPD explicitly rejects that view by treating the attention structure among local regions as a distillation target in its own right [2508.15653].

2. Tokenization of BEV feature maps

TGPD operates on intermediate BEV feature maps from teacher and student. Let the teacher’s intermediate BEV feature map be $F_T \in \mathbb{R}^{C \times H \times W}$ and the student’s BEV feature map be $F_S \in \mathbb{R}^{C \times H \times W}$. In the reported instantiation on nuScenes with BEV range $60 \times 30$ m and voxel resolution $0.15$ m, $(H, W) \simeq (400, 200)$ and $C = 256$ [2508.15653].

The first step is 2D patch extraction. A BEV feature map $F$ is partitioned into non-overlapping $p \times p$ cells, and the $i$th patch is defined as
$$
P_i(F) = \mathrm{flatten}!\left(F[:, (i_h \cdot p):(i_h \cdot p + p - 1), (i_w \cdot p):(i_w \cdot p + p - 1)]\right) \in \mathbb{R}^{p^2 \cdot C},
$$
where $(i_h, i_w)$ indexes the patch and the total patch count is
$$
N = (H/p) \cdot (W/p).
$$

Each patch vector is then linearly projected into a token embedding:
$$
p_i = W_e \cdot P_i(F) \in \mathbb{R}^d.
$$
TGPD also constructs a scene-level global token. The spatial average of $F$ is
$$
g = \frac{1}{H W}\sum_{h=1}^H \sum_{w=1}^W F[:, h, w] \in \mathbb{R}^C,
$$
which is projected as
$$
\tilde{g} = W_e \cdot g \in \mathbb{R}^d.
$$
For each model, the token sequence is therefore
$$
T = [\tilde{g}; p_1; p_2; \dots; p_N] \in \mathbb{R}^{(N+1)\times d}.
$$
This tokenization step is central to TGPD’s design because it converts dense BEV tensors into a structured set of local and global descriptors suitable for relational supervision [2508.15653].

3. Attention-guided patch selection

The distinctive mechanism in TGPD is its use of attention distributions as soft patch selectors. On each token set, TGPD applies one Transformer-style attention layer:
$$
Q = T W_Q, \qquad K = T W_K, \qquad V = T W_V,
$$
and
$$
A = \mathrm{softmax}!\left(\frac{Q K^T}{\tau}\right) \in \mathbb{R}^{(N+1)\times(N+1)},
$$
where $\tau$ is the temperature [2508.15653].

The attention matrix $A$ encodes how each patch token and the global token attend to all others. TGPD treats these attention distributions as indicators of importance in the BEV representation. In the wording of the paper, the central insight is to use the attention distributions themselves as “soft” selectors that highlight which patches the teacher deems important in the BEV. Distillation therefore targets not only local feature values but also the pattern of inter-patch attentions [2508.15653].

This design has two immediate consequences. First, it preserves long-range dependencies, including relations such as object-to-lane relations, because every token can attend to every other token. Second, it turns supervision into a focus-aware signal: the student is encouraged to reproduce the teacher’s allocation of attention rather than merely its raw activation magnitudes. A plausible implication is that this can better tolerate modality gaps than direct full-map regression, because the supervisory signal is organized around relational saliency rather than exhaustive pixelwise equivalence.

4. Loss formulation

TGPD combines two terms: a KL-divergence term on token-attention and an MSE term on local patch features. Let $A^T$ and $A^S$ be the attention matrices for teacher and student, respectively, after temperature scaling. The KL term
$$
\mathrm{KL}(A^S \parallel A^T)
$$
penalizes discrepancies in where the student “looks” in its BEV representation [2508.15653].

Using the notation from the paper, with $P(F)={P_i(F)}_{i=1\ldots N}$ as the patch-extraction operator, $g(F)=\frac{1}{HW}\sum F$ as the global-average operator, $W_e \in \mathbb{R}^{d\times(p^2\cdot C)}$, $W_Q, W_K, W_V \in \mathbb{R}^{d\times d}$ as learned projections, and $\lambda$ as the weighting factor on the MSE patch-feature term, the teacher-to-student TGPD loss is
$$

L_{T \to S}^{TGPD}

\mathrm{KL}(A^S \parallel A^T)
+
\lambda \cdot \sum_{i=1}^{N} \left| p_i(F_S) - p_i(F_T) \right|_2^2.
$$

The coach-to-student branch has the analogous form:
$$

L_{C \to S}^{TGPD}

\mathrm{KL}(A^S \parallel A^C)
+
\lambda \cdot \sum_{i=1}^{N} \left| p_i(F_S) - p_i(F_C) \right|_2^2.
$$

The overall TGPD objective is
$$

L_{TGPD}

\beta_1 L_{T \to S}^{TGPD}
+
\beta_2 L_{C \to S}^{TGPD},
$$
where $\beta_1$ and $\beta_2$ are the weights for the teacher-to-student and coach-to-student branches [2508.15653].

This formulation makes TGPD a multi-branch distillation mechanism inside the broader TCS framework. Factually, it aligns the student simultaneously to the teacher and the coach; contextually, this means that the coach functions as an intermediate source of supervision that reduces the cross-modal transfer gap.

5. Instantiation and implementation details

The reported implementation uses patch size $p = 16$, token embedding dimension $d = 256$, temperature $\tau = 0.2$, MSE weight $\lambda = 1.0$, and TGPD weights $\beta_1 = 0.6$, $\beta_2 = 0.4$, with the latter selected via ablation [2508.15653]. For $H = 400$ and $W = 200$, this yields approximately
$$
N \simeq (H/16)\cdot(W/16)\simeq 25\cdot 12 = 300
$$
patches.

The attention module is implemented as a single-layer multi-head attention block with $8$ heads, and $W_e$ is shared across teacher, coach, and student. All TGPD losses are computed only during training. The reported computational overhead is that tokenization plus one-layer attention adds $\lesssim 10\%$ training time and zero cost at inference, since TGPD is used only during training [2508.15653].

The paper also reports practical observations about optimization. TGPD generally accelerates student convergence, with the student reaching $34$ mIoU in $\sim 10$ epochs versus $\sim 30$ for the baseline. A warm-up of $\beta_1,\beta_2$ from $0 \to$ target over the first $5$ epochs stabilizes training. For transfer to other domains, the description recommends always including a global token to capture scene-level context, tuning $\tau$ between $0.1$ and $0.5$, and, when distilling from very high-capacity teachers, considering $\lambda < 1$ to avoid over-constraining local features [2508.15653]. These are reported as practical tips rather than universal guarantees.

6. Empirical behavior on nuScenes

The reported quantitative evidence is given on nuScenes for a camera-only HDMapNet student with a 30-epoch baseline of $31.16$ mIoU and $23.13$ mAP [2508.15653]. TGPD alone improves this baseline by $+4.84$ mIoU and $+8.23$ mAP, reaching $36.00$ mIoU and $31.36$ mAP. Combining TGPD with MSRD, even without the coach, yields $37.40$ mIoU and $31.90$ mAP. Within the full Teacher-Coach-Student framework, adding TGPD alone yields $37.73$ mIoU and $32.29$ mAP [2508.15653].

Setting	mIoU	mAP
Camera-only HDMapNet student, 30-epoch baseline	31.16	23.13
TGPD alone	36.00	31.36
TGPD + MSRD, without the coach	37.40	31.90
Full TCS framework, adding TGPD alone	37.73	32.29

These results are presented in the source as evidence that TGPD is by far the single most effective component for aligning cross-modal BEV features [2508.15653]. Interpreted conservatively, the pattern suggests that attention-guided local alignment accounts for a substantial fraction of the student’s recovery of teacher-side performance, and that the gain is not confined to one particular supervision path: it remains strong both in direct teacher-student distillation and in the full TCS setting.

7. Significance within MapKD

In MapKD as a whole, the student model is improved by $+6.68$ mIoU and $+10.94$ mAP while simultaneously accelerating inference speed [2508.15653]. TGPD is one of the two targeted distillation strategies introduced to support that outcome, the other being Masked Semantic Response Distillation (MSRD). TGPD’s specific function is BEV feature alignment; MSRD is described as semantic learning guidance. The distinction matters because TGPD addresses intermediate relational structure rather than terminal semantic responses alone [2508.15653].

Conceptually, TGPD occupies a specific niche within distillation for autonomous driving. It does not replace multimodal teachers, prior knowledge, or coach-based transfer; rather, it provides a mechanism for making their BEV representations legible to a vision-centric student. Its structure—patch tokenization, a global token, self-attention, KL alignment of attention matrices, and a light patchwise MSE term—defines a focused recipe for cross-modal BEV distillation. A plausible implication is that the method is relevant beyond the original MapKD setting wherever the teacher contains sensing modalities or priors unavailable to the student but the transfer target remains a spatially organized BEV representation [2508.15653].