TDDA: Top-Down Attention Deformable Aggregator

• TDDA is a feature-aggregation module that uses top-down semantic guidance via the class token to deform ROI pooling and capture adaptive region features.
• It integrates fixed multi-scale pyramid partitioning with deformable pooling to robustly handle irregular scene structures and viewpoint variations.
• TDDA employs dense offset prediction, GeM pooling, and hierarchical fusion to improve retrieval accuracy in visual place recognition.

Top-Down-attention-based Deformable Aggregator (TDDA) is the feature-aggregation module introduced in D$^2$-VPR for visual place recognition. It converts the backbone-plus-DRM output into a region-structured representation by combining fixed multi-scale pyramid partitioning with deformable region pooling guided by global semantics. In the formulation reported for D$^2$-VPR, TDDA uses the class token as a top-down controller, predicts a dense deformation field over the spatial feature map, warps the sampling grid inside each predefined ROI, applies GeM pooling to the deformed region features, performs hierarchical fusion across scales, injects deformation-aware positional information, and outputs region descriptors for a downstream cross-image transformer encoder (Zhang et al., 16 Nov 2025).

1. Definition and motivation

TDDA was proposed in the context of visual place recognition, where image aggregation based on fixed grids or rigid ROIs was described as insufficiently adaptive to irregular scene structure. The motivating claim is that many existing aggregators are “centered on spatial pooling” and therefore have weaker adaptability to irregular structures such as building facades, bridges, street layouts, skylines, partial occlusions, and viewpoint-induced distortions. TDDA addresses this by “dynamically and adaptively adjust[ing] the Regions of Interest (ROI) used for aggregation,” with the stated goal of better focusing on irregular geometric regions and improving robustness to viewpoint, scale, and structural variations (Zhang et al., 16 Nov 2025).

Within that formulation, the three components of the name are literal. “Top-down attention” refers to the use of high-level global semantic information—specifically the class token—to guide lower-level spatial aggregation. “Deformable” refers to the fact that sampling does not remain confined to a fixed ROI grid; instead, the sampling grid within each ROI is warped by learned offsets and scale adjustments. “Aggregator” refers to the sequence of region sampling, pooling, and multi-scale fusion used to produce the descriptor.

The module is not described as a generic transformer attention block. The paper instead characterizes it as an aggregation mechanism that preserves a structured multi-scale partition while making the sampling geometry adaptive. This distinction matters: TDDA does not discard the original pyramid partitioning strategy, but begins from fixed regions and then deforms the sampling pattern inside them.

2. Placement in the D$^2$-VPR pipeline

In D$^2$-VPR, TDDA is the third major block after the student backbone and the Distillation Recovery Module (DRM). The reported end-to-end feature flow is: input image, student backbone based on DINOv2-small ViT, DRM, reshape into a spatial feature map and a class token, TDDA, concatenation of region descriptors, cross-image transformer encoder following CricaVPR, flattening, and L2 normalization to obtain the final global descriptor (Zhang et al., 16 Nov 2025).

The upstream teacher-student setting determines TDDA’s input representation. The teacher is CricaVPR with DINOv2-base, and the student is D$^2$-VPR with DINOv2-small. The student’s final-layer tokens are

$X_4 \in \mathbb{R}^{B \times (1+P) \times D/2},$

while DRM concatenates features from all four student stages and linearly projects them into the teacher-dimensional space to produce

$$X_{\text{student}} \in \mathbb{R}^{B \times (1+P) \times D}.$$

TDDA then consumes this DRM-aligned output after it is rearranged into a patch-token feature map

$F \in \mathbb{R}^{B \times D \times H \times W}$

and a class token

$f_{\text{class}} \in \mathbb{R}^{B \times D}.$

The module therefore operates neither on raw backbone tokens nor on the teacher representation directly. Its direct inputs are the DRM-aligned dense spatial feature map and the associated global semantic token. This coupling to DRM is central to the D$^2$-VPR design, because the paper preserves the teacher-compatible multi-scale regional structure while introducing adaptive region deformation.

3. Internal structure and mathematical formulation

TDDA is divided into four submodules: Multi-Scale Pyramid ROI, Top-down Deformable Region Pooling, Hierarchical Down-top Fusion, and Deformation-Aware Position Embedding (Zhang et al., 16 Nov 2025).

Submodule	Main operation	Reported role
Multi-Scale Pyramid ROI	$^2$0 fixed regions	Structured regional partition
Top-down Deformable Region Pooling	Predict dense offsets/scales and warp ROI grid	Adaptive region extraction
Hierarchical Down-top Fusion	Aggregate lower-level pooled features into higher-level ones	Cross-scale reinforcement
Deformation-Aware Position Embedding	Add projected deformation information	Recover geometry after deformation

The Multi-Scale Pyramid ROI stage defines 14 regions in total: 1 global region, 4 medium-scale regions, and 9 fine-grained small regions. Each region is a rectangular ROI $^2$1 on the backbone feature map. The initial ROI layout is fixed, but the sampling within each ROI is subsequently deformed.

Top-down guidance is implemented by broadcasting the class token spatially and concatenating it with the dense feature map: $^2$2 with

$^2$3

A deformable generator containing two CNN layers predicts a dense four-channel deformation field: $^2$4 where

$^2$5

The four channels encode horizontal offset $^2$6, vertical offset $^2$7, raw width scaling $^2$8, and raw height scaling $^2$9.

For each ROI, a regular base grid

$^2$0

is constructed, and the deformation parameters are sampled at those grid points using bilinear interpolation: $^2$1 with

$^2$2

This gives each sampling location within the ROI its own local transform parameters $^2$3, so the deformation is spatially varying inside the ROI rather than a single box-level transform.

The warped coordinates are defined as

$^2$4

where $^2$5 are normalized base-grid coordinates, $^2$6 is the ROI center, and $^2$7 are the ROI width and height. The deformed grid $^2$8 is then used to sample the original feature map: $^2$9 where

$^2$0

Each deformed region feature is compressed by GeM pooling: $^2$1 where $^2$2 is a learnable exponent. The module then introduces hierarchical down-top fusion: $^2$3 in which a higher-level region is enhanced by the average of associated lower-level neighboring regions after linear projection. The paper states that neighboring relations are defined based on deformed region centers.

Because deformable pooling disrupts the original regular geometry, TDDA also injects deformation-aware positional information: $^2$4 The paper states this formula but does not fully specify the tensor-shape reduction needed to map $^2$5 to a vector compatible with $^2$6. The intended role is nonetheless explicit: add post-deformation geometric information to the pooled descriptor.

After pooling, fusion, and positional enhancement, all region descriptors are concatenated into a unified sequence and sent to a cross-image transformer encoder; the output sequence is flattened and L2-normalized to form the final descriptor.

4. Relation to deformable-attention research

TDDA belongs to a broader family of sparse, geometry-aware aggregation mechanisms, but its exact implementation differs from canonical deformable attention. The most direct technical precedent is Deformable DETR, which replaced dense global attention with sparse sampling around reference points. Its single-scale operator is

$^2$7

and its multi-scale extension is

$^2$8

That work established the reusable primitives of sparse reference-point sampling, query-dependent offsets, bilinear interpolation, and multi-scale aggregation, and explicitly framed deformable attention as a more efficient and detection-aligned alternative to dense attention over image lattices (Zhu et al., 2020).

TDDA, however, is not identical to MSDeformAttn. It does not describe attention heads, normalized attention weights over sampled points, or query-conditioned pointwise value aggregation in the transformer sense. Its mechanism is more accurately characterized, in the paper’s own terms, as top-down-guided deformable ROI pooling via dense field prediction and grid warping. The class token plays the semantic guidance role, while the dense deformation field supplies the local geometric adaptation.

A second relevant comparison is the multi-object pose estimation model based on Multi-Resolution Deformable Attention and Query Aggregation. That work uses a deformable multi-resolution operator,

$^2$9

to sample sparse locations across multiple feature resolutions, and supplements it with a post-decoder query aggregation scheme based on grouping and concatenation. It is strongly related on the “deformable” and “aggregation” axes, but it does not introduce an explicit top-down feature-guidance mechanism of the kind used in TDDA (Periyasamy et al., 2023).

A third comparison is TopoBDA, which operates over Bird’s Eye View features and uses Bezier control points as deformable attention anchors: $^2$0 That design is top-down in the spatial sense of BEV and geometry-aware in the sense of curve-conditioned sampling. It is not TDDA by name, but it is a close proxy for a task-specific top-down deformable aggregator specialized to elongated lane structures (Kalfaoglu et al., 2024).

Taken together, these comparisons place TDDA in a specific niche: it is closer to adaptive deformable region pooling than to transformer cross-attention, while inheriting the general sparse-sampling intuition that deformable attention introduced.

5. Empirical evidence and efficiency profile

The principal direct evidence for TDDA comes from the ablation study on MSLS-val. The reported baseline obtains $^2$1 for R@1 / R@5 / R@10. Adding DRM yields $^2$2. Adding TDDA on top of the DRM-enhanced baseline yields $^2$3. The paper interprets this as evidence that TDDA can “flexibly handle irregular ROIs and enhance overall retrieval accuracy” (Zhang et al., 16 Nov 2025).

The reported parameter and compute profile also situates TDDA within the full D$^2$4-VPR system. The complete model with encoder has $^2$5M parameters and $^2$6 GFLOPs, compared with CricaVPR’s $^2$7M parameters and $^2$8 GFLOPs. The overall framework is stated to reduce parameter count by about $^2$9 and FLOPs by about $X_4 \in \mathbb{R}^{B \times (1+P) \times D/2},$0. Within that system-level budget, TDDA itself has $X_4 \in \mathbb{R}^{B \times (1+P) \times D/2},$1M parameters, whereas the cross-image encoder has $X_4 \in \mathbb{R}^{B \times (1+P) \times D/2},$2M parameters. The model outputs $X_4 \in \mathbb{R}^{B \times (1+P) \times D/2},$3-dimensional global features before PCA.

These numbers support a specific interpretation. TDDA is not the sole source of the overall efficiency gain, since D$X_4 \in \mathbb{R}^{B \times (1+P) \times D/2},$4-VPR also relies on a smaller student backbone and distillation, but its own parameter footprint is comparatively small relative to the full network. At the same time, the paper explicitly identifies a practical limitation: “The deformable aggregator introduces sampling operations that constrain inference speed.” The algorithmic adaptation afforded by deformable sampling therefore comes with a hardware-efficiency tradeoff.

6. Scope, interpretation, and limitations

TDDA should not be conflated with several adjacent ideas. It is not standard GeM pooling, because GeM is applied only after deformable region extraction. It is not deformable convolution in the usual sense, because the paper formulates the mechanism around ROI-wise grid warping and GridSample. It is not transformer-style deformable attention in the strict Deformable DETR sense, because the paper does not present headwise attention-weight normalization over sparse sampled keys. It is also not a classical FPN-style top-down pathway; “top-down” here means semantic guidance from the global class token to local region deformation (Zhang et al., 16 Nov 2025).

The design nevertheless contains a bidirectional logic. First, global semantics guide local sampling through top-down fusion of the class token with the feature map. Second, local deformed regions reinforce larger-scale descriptors through hierarchical down-top fusion. This suggests a deliberately asymmetric information flow: semantic conditioning downward, then feature reinforcement upward. A plausible implication is that the module is intended to retain structured regional priors while relaxing the rigidity of fixed partitions.

Several implementation details remain unspecified in the available description. The paper does not provide numerical values for $X_4 \in \mathbb{R}^{B \times (1+P) \times D/2},$5, $X_4 \in \mathbb{R}^{B \times (1+P) \times D/2},$6, $X_4 \in \mathbb{R}^{B \times (1+P) \times D/2},$7, or $X_4 \in \mathbb{R}^{B \times (1+P) \times D/2},$8; does not specify the activation functions or normalization layers inside the deformable generator; does not report an initialization strategy for offsets and scales; does not state whether $X_4 \in \mathbb{R}^{B \times (1+P) \times D/2},$9 and $$X_{\text{student}} \in \mathbb{R}^{B \times (1+P) \times D}.$$0 are constrained by a particular nonlinearity; and does not fully explain how $$X_{\text{student}} \in \mathbb{R}^{B \times (1+P) \times D}.$$1 is reduced and projected for deformation-aware positional embedding. It also reports no TDDA-specific auxiliary loss: training is driven by distillation-stage MSE and fine-tuning multi-similarity loss on the final descriptors.

For that reason, TDDA is best understood as a task-specific aggregation module with a clearly specified conceptual structure but only partially specified low-level implementation. Its distinctive contribution is not a new universal deformable-attention formalism, but a concrete VPR-oriented synthesis: preserve a multi-scale region pyramid, use the class token as a semantic controller, deform ROI sampling geometries through a dense predicted field, pool the resulting irregular regions, and fuse them into a descriptor suited to place recognition.