Adaptive Semantic Aggregation Module

• Adaptive Semantic Aggregation (ASA) is a neural module that adaptively aggregates patch-level features using attention-weighted clustering to form semantically coherent part-level representations.
• It integrates as an add-on to Vision Transformer architectures, significantly enhancing cross-view image retrieval performance in UAV-satellite matching through soft partitioning.
• The module employs nonparametric clustering and soft assignment, providing robust, adaptable part-level discrimination without additional parameters or complex supervision.

Adaptive Semantic Aggregation (ASA) is a neural module designed for part-level feature modeling in visual geo-localization, particularly under cross-view conditions such as unmanned aerial vehicle (UAV) and satellite image matching. ASA operates as an adaptive, attention-weighted aggregator of patch-level transformer features, constructing semantically coherent regional representations that support robust image retrieval under significant viewpoint and scale variations. The algorithmic core systematically associates image patches with a learned, per-image set of “part” features by clustering on interpretable, scalar semantic proxies, followed by a soft assignment scheme that enables semantic awareness and adaptivity at inference.

1. Module Position and Architectural Integration

ASA is implemented as an add-on to the Vision Transformer (ViT) encoder, following its last transformer block. The ViT encoder outputs a global “class” token $x_{\mathrm{cls}}\in\mathbb{R}^{1\times D}$ and $N$ patch-level embeddings $\{P_i\in \mathbb{R}^{1\times D}\}_{i=1}^N$. The ASA module consumes the patch embeddings $P = [P_1;P_2;\cdots;P_N]\in\mathbb{R}^{N\times D}$ and outputs $K$ part-level features $\{\rho_k\in\mathbb{R}^{1\times D}\}_{k=1}^K$. Both the original class token and these part features are subsequently processed by independent additive transformation layers and classification heads. During inference, the transformed global and part features are concatenated for retrieval.

2. Mathematical Formulation and Aggregation Workflow

The ASA module proceeds in the following steps:

1. Semantic Proxy for Clustering

For each patch embedding $P_i\in\mathbb{R}^{1\times D}$, a scalar proxy,

$Q_i = \frac{1}{D}\sum_{d=1}^{D}P_i^{\,d},$

is computed. The sequence $Q = [Q_1,\dots,Q_N]\in\mathbb{R}^{N\times 1}$ serves as a low-complexity, 1-D semantic representation used to initialize clustering.

2. Anchor (Prototype) Initialization and K-Means Refinement

Patches are sorted in descending order by $Q_i$, and cluster center indices are initialized as $IC_k = \frac{(2k-1)\,N}{2K},\;\;\; k=1,\dots,K$. Anchors $P_{C_k}=P_{S_{IC_k}}$ are then defined for each $k$. A limited number of k-means steps are run on $\{Q_i\}$ for further refinement, always using the original patch embeddings as anchor representations.

3. Patch-to-Anchor Distance Calculation

The $\ell_2$ distance between each patch and each anchor is computed:

$$\mathrm{dis}_k^i = \left\lVert P_i - P_{C_k}\right\rVert_2,\;\;\; i = 1,\dots,N,\; k = 1,\dots,K.$$

4. Soft Attention Weighting

Distances are normalized within each part $k$ and mapped to attention weights:

$$A_k^i = \cos\left( \frac{\mathrm{dis}_k^i - \mathrm{dis}_k^{\min}}{\mathrm{dis}_k^{\max} - \mathrm{dis}_k^{\min}} \cdot \frac{\pi}{2} \right).$$

Here, $\alpha=1$, $\beta=0$ (default), with $\mathrm{dis}_k^{\min}=\min_i\mathrm{dis}_k^i$ and $\mathrm{dis}_k^{\max}=\max_i\mathrm{dis}_k^i$.

5. Adaptive Semantic Aggregation

Each part feature is given by the attention-weighted average of all patch features:

$$\rho_k = \frac{\sum_{i=1}^N A_k^i\,P_i}{\sum_{i=1}^N A_k^i},\quad k=1,\dots,K,\quad \rho_k \in \mathbb{R}^{1\times D}.$$

This mechanism supports a semantically “soft” region-to-patch assignment, distinguishing it from hard clustering approaches.

3. Training Paradigm and Supervision

ASA’s parameterization is non-parametric with respect to the clustering/aggregation stage, with part prototypes $\{P_{C_k}\}$ re-identified via the clustering procedure each forward pass, thus adapting to the statistics of each input image.

Supervision is two-fold:

• Cross-Entropy Loss across $K+1$ classification heads (one global and $K$ local):

$$L_{\mathrm{CE}} = -\frac{1}{K+1}\sum_{k=0}^{K}\log\frac{\exp(z_k(y))}{\sum_{c=1}^C\exp(z_k(c))}$$

where $z_k$ are the logits for class $k$ and $C$ is the number of locations.

• Triplet Loss on the features before classification:

$$L_{\mathrm{Triplet}}=\frac{1}{K+1}\sum_{k=0}^K \max\left(d(f_k, f_k^+) - d(f_k, f_k^-)+M, 0\right)$$

with $M=0.3$, $d(\cdot, \cdot)$ the distance metric, and $f_k^+, f_k^-$ positive and negative samples.

The total loss is $$L_{\mathrm{total}} = L_{\mathrm{CE}} + L_{\mathrm{Triplet}}$$.

4. Integration in UAV Visual Geo-localization

ASA is employed in a two-branch (siamese) ViT system, processing UAV and satellite images with shared weights. Both branches output global and $K$ part-level features, each supervised independently as described above. During testing, the final image representation is

$[\; f_0\ ;\ f_1\ ;\ \ldots\ ;\ f_K\ ]$

of dimension $(K+1)\times D$. Retrieval between query (UAV) and gallery (satellite) images is performed via Euclidean distance.

Ablation results on University-1652 demonstrate ASA’s effectiveness over alternative part representation strategies (see Section 6). The adaptive, soft attention mechanism underlying ASA enables improved semantic partitioning and part-level discrimination, which is critical for cross-view matching where spatial alignment is nontrivial.

5. Quantitative Performance and Ablation Analyses

On University-1652, using a ViT-S backbone at $256\times256$ input resolution, the following Recall@1 (R@1) and Average Precision (AP) metrics were reported:

Method	UAV→Sat R@1 / AP	Sat→UAV R@1 / AP
FSRA (hard partition)	84.51 / 86.71	88.45 / 83.37
ASA (soft partition)	85.12 / 87.21	89.30 / 84.17

Ablation on partition strategies (UAV→Sat):

Partition Strategy	Recall@1	AP
Uniform hard	83.98	86.27
K-means hard	84.97	87.12
K-means soft (ASA)	85.12	87.21

Ablation on number of parts $K$ (UAV→Sat, R@1):

$K$	Recall@1
1	72.11
2	85.12
3	84.73
4	84.48

These results show that soft, attention-based partitioning via ASA consistently yields superior retrieval performance with an optimal $K=2$. The benefit of differentiable soft assignment over hard or uniform splits is empirically confirmed by absolute gains of $0.6\%$–$1.0\%$ in Recall@1 relative to prior state-of-the-art (Li et al., 2024).

6. Distinctive Properties and Implications

The primary distinguishing feature of ASA is its semantic adaptivity. By clustering on scalar semantic proxies and enabling soft, attention-based aggregation, ASA avoids hard spatial partitioning and can accommodate ambiguous or non-rigid part boundaries, which is vital in UAV–satellite image matching under severe viewpoint and scale variation. The nonparametric nature of prototype/anchor selection and the per-image reinterpretation of patch-to-part associations underpin robust generalization and makes the module naturally compatible with transformer-based architectures.

A plausible implication is that ASA’s mechanism could be deployed beyond aerial geo-localization in any vision domain where part-level semantic compositionality and viewpoint/scale invariance are necessary, provided transformer backbones are used.

7. Context and Significance in Part-level Representation Research

Previous part-based approaches for geo-localization, such as FSRA (Fixed Spatial Region Aggregation), predominantly relied on hard partitioning—either uniform or heuristic—limiting their semantic expressivity and adaptivity. The introduction of ASA addresses these limitations by leveraging per-image, feature-driven partitioning and soft assignments, resulting in more semantically meaningful regional features. Quantitative improvements were achieved without additional model parameters or auxiliary supervision. As a result, ASA represents a notable methodological advance in part-level representation learning for cross-view image retrieval (Li et al., 2024).