Task Modulated Deformable Attention (TMDA)

• The paper introduces TMDA to split the decoder query into classification-aware and localization-aware branches, addressing feature misalignment.
• It employs shared offset learning with task-specific attention weights, resulting in improved mAP scores (from 60.4 to 61.6) on nuScenes.
• TMDA is integrated into DAMap’s DETR-like pipeline for online HD map construction, balancing efficient geometric sampling with enhanced task-specific features.

Task Modulated Deformable Attention (TMDA) is a decoder-side deformable cross-attention mechanism introduced in DAMap for online HD map construction from multi-view surround-view RGB images. In that setting, the model predicts vectorized map elements such as road boundaries, lane dividers, and pedestrian crossings, and the central motivation for TMDA is task misalignment between classification and localization. TMDA addresses the feature-side part of that misalignment by replacing a task-shared deformable attention process with a design that splits the decoder query into classification-aware and localization-aware branches, keeps offset learning shared, and learns task-specific attention weights so that the two tasks receive different attended features (Dong et al., 26 Oct 2025).

1. Problem formulation and motivation

DAMap studies online HD map construction under the requirement that predictions have both high classification quality and high localization quality. The paper argues that existing methods perform poorly on high-quality predictions because of two defects: inappropriate task labels under one-to-many matching, and sub-optimal task features caused by a task-shared sampling mechanism (Dong et al., 26 Oct 2025).

TMDA is aimed at the second defect. The paper’s diagnosis is that standard decoder design uses the same instance features for both classification and localization, even though the two tasks prefer different evidence. Its concrete example is that classification prefers semantically salient regions inside an instance, whereas localization prefers boundary-sensitive or geometry-sensitive regions. When conventional deformable attention extracts one shared instance feature and feeds it to both heads, the resulting representation can create inherent conflicts between tasks and become sub-optimal for both outputs (Dong et al., 26 Oct 2025).

Within this framing, TMDA is not a general synonym for any multitask deformable attention module. In DAMap, the term has a specific meaning: the deformable cross-attention layer is modified so that classification and localization no longer share the same attention weighting over sampled features, even though they still share the same sampled neighborhood. The paper also refers to the same module as Task Decoupled Deformable Attention, and the two names denote the same proposed mechanism (Dong et al., 26 Oct 2025).

2. Formal mechanism

DAMap places TMDA in the decoder after self-attention. The decoder query is first doubled in channel dimension at initialization, and after self-attention it is split into two task-aware queries:

$$\mathbf{Q}_{\mathrm{cls}}, \mathbf{Q}_{\mathrm{loc}} = \mathrm{Split}(\mathrm{SA}(\mathbf{Q})),$$

with the dimensions stated as $\mathbf{Q}\in\mathbb{R}^{2C}$ and $$\mathbf{Q}_{\mathrm{cls}}, \mathbf{Q}_{\mathrm{loc}}\in\mathbb{R}^{C}$$ (Dong et al., 26 Oct 2025).

The paper presents a simplified reference form of standard deformable attention as

$$\mathbf{A} = W_a \mathbf{Q}, \qquad \Delta r = W_p \mathbf{Q}, \qquad \mathbf{V} = \mathrm{Samp}(W_v \mathbf{F}, r+\Delta r),$$

$$\mathrm{DeformableAttention}(\mathbf{Q},\mathbf{V}) = \mathrm{Softmax}(\mathbf{A})\mathbf{V}.$$

Here, a single query determines both the offsets $\Delta r$ and the attention weights $\mathbf{A}$, and therefore both tasks are forced to use the same sampled evidence and the same aggregation rule (Dong et al., 26 Oct 2025).

TMDA changes this by learning task-specific attention weights while keeping offset learning shared:

$$\mathbf{A}_{\mathrm{cls}} = W_a \mathbf{Q}_{\mathrm{cls}}, \qquad \mathbf{A}_{\mathrm{loc}} = W_a' \mathbf{Q}_{\mathrm{loc}},$$

$$\Delta r = W_p \,\mathrm{Cat}(\mathbf{Q}_{\mathrm{cls}}, \mathbf{Q}_{\mathrm{loc}}), \qquad \mathbf{V} = \mathrm{Samp}(W_v\mathbf{F}, r+\Delta r),$$

$$\hat{\mathbf{Q}}_{\mathrm{cls}} = \mathrm{Softmax}(\mathbf{A}_{\mathrm{cls}})\mathbf{V}, \qquad \hat{\mathbf{Q}}_{\mathrm{loc}} = \mathrm{Softmax}(\mathbf{A}_{\mathrm{loc}})\mathbf{V}.$$

The outputs $\mathbf{Q}\in\mathbb{R}^{2C}$0 and $\mathbf{Q}\in\mathbb{R}^{2C}$1 are the classification-specific and localization-specific attended features, respectively (Dong et al., 26 Oct 2025).

This decomposition is exact about what is shared and what is task-specific. Shared components include the BEV feature projection $\mathbf{Q}\in\mathbb{R}^{2C}$2, the sampled value tensor $\mathbf{Q}\in\mathbb{R}^{2C}$3, the offset projection $\mathbf{Q}\in\mathbb{R}^{2C}$4, and the sampling locations $\mathbf{Q}\in\mathbb{R}^{2C}$5. Task-specific components include the task-aware queries $\mathbf{Q}\in\mathbb{R}^{2C}$6 and $\mathbf{Q}\in\mathbb{R}^{2C}$7, the attention projections $\mathbf{Q}\in\mathbb{R}^{2C}$8 and $\mathbf{Q}\in\mathbb{R}^{2C}$9, and the resulting attention weights $$\mathbf{Q}_{\mathrm{cls}}, \mathbf{Q}_{\mathrm{loc}}\in\mathbb{R}^{C}$$0 and $$\mathbf{Q}_{\mathrm{cls}}, \mathbf{Q}_{\mathrm{loc}}\in\mathbb{R}^{C}$$1 (Dong et al., 26 Oct 2025).

A crucial negative definition is equally important: TMDA is not implemented as two completely separate deformable attention layers. DAMap explicitly states that it does not simply use two cross-attention layers to process task-aware queries separately. Instead, it uses one module in which offset learning remains shared while attention weights are task-specific, because learning task-specific weights and task-specific offsets simultaneously is described as difficult, and offset learning is described as harder because its objective is unbounded (Dong et al., 26 Oct 2025).

3. Architectural placement within DAMap

DAMap follows a standard DETR-like online map construction pipeline. Multi-view images are processed by a shared backbone, image features are transformed into bird’s-eye-view features, and the resulting BEV tensor is denoted

$$\mathbf{Q}_{\mathrm{cls}}, \mathbf{Q}_{\mathrm{loc}}\in\mathbb{R}^{C}$$2

A transformer decoder then predicts map element instances (Dong et al., 26 Oct 2025).

Within that decoder, TMDA belongs specifically to the cross-attention stage. The paper writes the decoder context as

$$\mathbf{Q}_{\mathrm{cls}}, \mathbf{Q}_{\mathrm{loc}}\in\mathbb{R}^{C}$$3

with $$\mathbf{Q}_{\mathrm{cls}}, \mathbf{Q}_{\mathrm{loc}}\in\mathbb{R}^{C}$$4 the decoder queries, $$\mathbf{Q}_{\mathrm{cls}}, \mathbf{Q}_{\mathrm{loc}}\in\mathbb{R}^{C}$$5 the BEV feature map, $$\mathbf{Q}_{\mathrm{cls}}, \mathbf{Q}_{\mathrm{loc}}\in\mathbb{R}^{C}$$6 self-attention, $$\mathbf{Q}_{\mathrm{cls}}, \mathbf{Q}_{\mathrm{loc}}\in\mathbb{R}^{C}$$7 cross-attention, and $$\mathbf{Q}_{\mathrm{cls}}, \mathbf{Q}_{\mathrm{loc}}\in\mathbb{R}^{C}$$8 the feed-forward network. TMDA replaces the conventional shared deformable cross-attention behavior in this stage, after which the task-specific attended features are passed to different FFN networks and then to different prediction heads (Dong et al., 26 Oct 2025).

The broader DAMap architecture contains three components with distinct roles. Distance-aware Focal Loss (DAFL) changes classification supervision for one-to-many matching samples; Hybrid Loss Scheme (HLS) applies standard Focal Loss in early decoder layers and DAFL in later decoder layers; TMDA changes decoder feature extraction so that classification and localization obtain different task-specific features. The paper describes these components as complementary and as addressing different issues in the HD map task (Dong et al., 26 Oct 2025).

TMDA itself is optimized indirectly through the downstream task losses rather than through a separate auxiliary objective. The paper does not define a standalone TMDA loss. Instead, its parameters are trained through the usual classification loss and localization loss used by the overall system (Dong et al., 26 Oct 2025).

The implementation description is deliberately limited. DAMap states that MapTRv2 typically stacks 6 decoder layers and presents TMDA as a decoder design change, but it does not give exact TMDA hyperparameters such as the number of attention heads, number of sampling points, FFN dimensions, or explicit per-layer insertion details. It also reports experiments primarily with ResNet-50 and additionally with Swin-B (Dong et al., 26 Oct 2025).

4. Empirical behavior and ablation evidence

The main direct evidence for TMDA is the component ablation on nuScenes validation. Under the reproduced MapTRv2 baseline, the baseline scores are $$\mathbf{Q}_{\mathrm{cls}}, \mathbf{Q}_{\mathrm{loc}}\in\mathbb{R}^{C}$$9, $$\mathbf{A} = W_a \mathbf{Q}, \qquad \Delta r = W_p \mathbf{Q}, \qquad \mathbf{V} = \mathrm{Samp}(W_v \mathbf{F}, r+\Delta r),$$0, $$\mathbf{A} = W_a \mathbf{Q}, \qquad \Delta r = W_p \mathbf{Q}, \qquad \mathbf{V} = \mathrm{Samp}(W_v \mathbf{F}, r+\Delta r),$$1, and $$\mathbf{A} = W_a \mathbf{Q}, \qquad \Delta r = W_p \mathbf{Q}, \qquad \mathbf{V} = \mathrm{Samp}(W_v \mathbf{F}, r+\Delta r),$$2. With TMDA only, the scores become $$\mathbf{A} = W_a \mathbf{Q}, \qquad \Delta r = W_p \mathbf{Q}, \qquad \mathbf{V} = \mathrm{Samp}(W_v \mathbf{F}, r+\Delta r),$$3, $$\mathbf{A} = W_a \mathbf{Q}, \qquad \Delta r = W_p \mathbf{Q}, \qquad \mathbf{V} = \mathrm{Samp}(W_v \mathbf{F}, r+\Delta r),$$4, $$\mathbf{A} = W_a \mathbf{Q}, \qquad \Delta r = W_p \mathbf{Q}, \qquad \mathbf{V} = \mathrm{Samp}(W_v \mathbf{F}, r+\Delta r),$$5, and $$\mathbf{A} = W_a \mathbf{Q}, \qquad \Delta r = W_p \mathbf{Q}, \qquad \mathbf{V} = \mathrm{Samp}(W_v \mathbf{F}, r+\Delta r),$$6. TMDA alone therefore improves mAP from $$\mathbf{A} = W_a \mathbf{Q}, \qquad \Delta r = W_p \mathbf{Q}, \qquad \mathbf{V} = \mathrm{Samp}(W_v \mathbf{F}, r+\Delta r),$$7 to $$\mathbf{A} = W_a \mathbf{Q}, \qquad \Delta r = W_p \mathbf{Q}, \qquad \mathbf{V} = \mathrm{Samp}(W_v \mathbf{F}, r+\Delta r),$$8, with especially visible gains on lane divider and boundary prediction (Dong et al., 26 Oct 2025).

When combined with the supervision-side components, DAMap reports $$\mathbf{A} = W_a \mathbf{Q}, \qquad \Delta r = W_p \mathbf{Q}, \qquad \mathbf{V} = \mathrm{Samp}(W_v \mathbf{F}, r+\Delta r),$$9, $$\mathrm{DeformableAttention}(\mathbf{Q},\mathbf{V}) = \mathrm{Softmax}(\mathbf{A})\mathbf{V}.$$0, $$\mathrm{DeformableAttention}(\mathbf{Q},\mathbf{V}) = \mathrm{Softmax}(\mathbf{A})\mathbf{V}.$$1, and $$\mathrm{DeformableAttention}(\mathbf{Q},\mathbf{V}) = \mathrm{Softmax}(\mathbf{A})\mathbf{V}.$$2. Relative to the same $$\mathrm{DeformableAttention}(\mathbf{Q},\mathbf{V}) = \mathrm{Softmax}(\mathbf{A})\mathbf{V}.$$3 baseline, the full system gains $$\mathrm{DeformableAttention}(\mathbf{Q},\mathbf{V}) = \mathrm{Softmax}(\mathbf{A})\mathbf{V}.$$4 mAP. This is the empirical basis for treating TMDA as complementary to DAFL and HLS rather than as a replacement for them (Dong et al., 26 Oct 2025).

A more diagnostic ablation compares several decoupling designs at the same parameter count of 52M. The baseline uses 40M parameters and obtains 60.4 mAP. Among the 52M variants, the design with task-specific sampling offsets and attention weights generated by task-aware query reaches 60.7 mAP; the design with task-specific sampling offsets and attention weights learned by task-agnostic query reaches 60.9; the design with task-specific sampling offsets and task-agnostic attention weights learned by task-agnostic query remains at 60.4; and the DAMap design with shared/task-agnostic offsets and task-specific attention weights reaches 61.6. This is the paper’s main empirical support for the specific shared-offset, task-specific-weight formulation (Dong et al., 26 Oct 2025).

The reported overhead is modest but nonzero. The reproduced MapTRv2 baseline runs at 12.6 FPS, whereas DAMap runs at 12.1 FPS. The same comparison reports a parameter increase from 40M to 52M. The paper therefore characterizes the inference overhead as small (Dong et al., 26 Oct 2025).

Supplementary analysis is consistent with the intended mechanism. The attention-weight visualization shows that the weight distribution for classification and localization is different, and that both differ from the weight distribution when tasks are shared. Additional score-distribution analyses show that true positives more often receive high scores and false positives more often receive low scores with DAMap, and that at classification score $$\mathrm{DeformableAttention}(\mathbf{Q},\mathbf{V}) = \mathrm{Softmax}(\mathbf{A})\mathbf{V}.$$5 the model has a better localization-quality distribution. These analyses are not isolated to TMDA alone, but they are consistent with TMDA’s stated role in reducing feature-side task misalignment (Dong et al., 26 Oct 2025).

5. Relation to adjacent deformable and multitask attention mechanisms

TMDA belongs to a broader family of work that combines deformable sampling with task interaction, but its mechanism is narrower and more explicit than several nearby designs.

Mechanism	Closest overlap with TMDA	Key mismatch
Deformable Inter-Task Self-Attention (Bohn et al., 6 Aug 2025)	Sparse deformable attention across concatenated task feature maps	No explicit task embeddings, task-conditioned parameters, task-specific reference points, or task-conditioned gating
DeMT (Xu et al., 2023)	Task-aware decoding over task-specific deformable features	Deformability is in the encoder via deformable convolution; decoder attention is standard MHSA
DAT++ (Xia et al., 2023)	Content-conditioned deformable sampling in a vision backbone	Task-agnostic attention module across downstream tasks
Deformable VisTR (Yarram et al., 2022)	Query-conditioned sparse deformable attention with learned offsets and weights	Spatio-temporal, content-conditioned, and not task-modulated

The multitask dense vision transformer work on Deformable Inter-Task Self-Attention is especially close in spirit. It replaces global inter-task self-attention with deformable sampling over concatenated multi-task and multi-scale feature maps, and task identity is encoded structurally through the arrangement of the concatenated task tensor and through task-specific query content. The paper is explicit, however, that it does not introduce explicit task embeddings, task-conditioned parameters, task-specific reference points, task-conditioned gating, or modulation coefficients; it is therefore better described as deformable inter-task attention than TMDA in a narrow sense (Bohn et al., 6 Aug 2025).

DeMT is a system-level precursor rather than a literal TMDA design. Its encoder uses a spatial-aware deformable operator implemented with deformable convolution to generate task-specific deformed features, while its decoder uses a task interaction block and a task query block based on standard multi-head self-attention. The strongest conceptual overlap is that each task uses its own deformable feature as a query over shared task-interacted memory, but the paper does not define a unified task-modulated deformable attention operator (Xu et al., 2023).

DAT++ provides a different point of comparison. It introduces deformable multi-head attention for general visual recognition by predicting a shared sparse set of image-dependent key/value locations from the current feature map. Its deformable attention is already feature-conditioned, stage-dependent, and group-dependent, but it is task-agnostic in parameterization; the same module is reused across classification, detection, and segmentation without explicit task conditioning (Xia et al., 2023).

Deformable VisTR extends deformable attention into 3D spatio-temporal coordinates for video instance segmentation. Its offsets and attention weights are predicted from the query feature alone, which makes it content-conditioned rather than task-conditioned. The relevance to TMDA lies in the shared structural idea of replacing dense all-to-all attention with sparse learned sampling, not in explicit task modulation (Yarram et al., 2022).

6. Interpretation, distinctions, and limitations

The central distinction in TMDA is that it decouples aggregation more than sampling. Classification and localization use the same sampled values $$\mathrm{DeformableAttention}(\mathbf{Q},\mathbf{V}) = \mathrm{Softmax}(\mathbf{A})\mathbf{V}.$$6 derived from the same offsets $$\mathrm{DeformableAttention}(\mathbf{Q},\mathbf{V}) = \mathrm{Softmax}(\mathbf{A})\mathbf{V}.$$7, but they apply different attention weights to those values. This suggests that the DAMap formulation assumes the two tasks often need to inspect roughly the same sampled neighborhood while assigning different importance to the samples within that neighborhood (Dong et al., 26 Oct 2025).

Several misconceptions are directly addressed by the paper. First, TMDA is not merely another name for generic multitask deformable attention. In the strict sense used by DAMap, TMDA means a decoder cross-attention design with task-aware query splitting, shared offset learning, and task-specific attention weights (Dong et al., 26 Oct 2025). Second, TMDA is not equivalent to duplicating deformable attention into two independent branches; the paper explicitly rejects that implementation and reports that the chosen shared-offset design performs better than the tested alternatives (Dong et al., 26 Oct 2025).

The module also has identifiable assumptions and trade-offs. Parameter count increases from 40M to 52M, and throughput decreases from 12.6 FPS to 12.1 FPS. A plausible implication is that TMDA buys representational decoupling at moderate architectural cost rather than for free. Another plausible implication is that shared offsets may be less expressive than fully separate task-specific offsets when the two tasks truly require different sample locations, although DAMap’s ablation indicates that fully separate offset-and-weight designs were harder to optimize and less effective in this setting (Dong et al., 26 Oct 2025).

In the current literature, TMDA is therefore best understood as a specific answer to a specific problem: feature-side task misalignment in DETR-like HD map construction. Its novelty is not deformable attention alone, and not task branching alone, but the particular compromise between the two—shared geometric sampling with task-specific weighting. The broader deformable-attention literature shows many ways to make attention sparse, adaptive, or cross-task, but DAMap makes the strongest explicit claim to task-modulated deformable attention in the narrow sense of a module whose modulation is tied directly to classification-aware and localization-aware query branches (Dong et al., 26 Oct 2025).