---
title: Dynamic Multi-level Weighted Alignment Network
url: https://www.emergentmind.com/topics/dynamic-multi-level-weighted-alignment-network
type: topic
---

# Dynamic Multi-level Weighted Alignment Network

Searching arXiv for the primary paper and closely related alignment works to ground the article in cited sources.
Dynamic Multi-level Weighted Alignment Network (DMWAN) is a zero-shot sketch-based image retrieval (ZS-SBIR) architecture that addresses two training-time deficiencies identified in prior work: the use of imbalanced samples of modalities and the presence of inconsistent low-quality information, both of which lead to sub-optimal performance. The model comprises three components: a Uni-modal Feature Extraction Module, a Cross-modal Multi-level Weighting Module, and a Weighted Quadruplet Loss Module. In the reported formulation, sketch and image inputs are encoded by a ViT, category text is encoded by a CLIP text encoder, alignment quality is estimated from both local and global cross-modal agreement, and the resulting dynamic weights modulate a quadruplet-style metric learning objective. Experiments on Sketchy, TU-Berlin, and QuickDraw are reported to show superior performance over state-of-the-art ZS-SBIR methods [2511.00925].

## 1. Problem formulation and scope

DMWAN is proposed for ZS-SBIR, where a sketch query must retrieve images from unseen classes at test time. The motivating claim is that previous methods suffer from two related issues during training: imbalanced samples of modalities and inconsistent low-quality information. The network is designed to mitigate both by assigning a sample-dependent alignment weight to each sketch-image pair and by incorporating that weight into a domain-balanced metric loss [2511.00925].

Within the broader alignment literature, DMWAN belongs to a family of methods that estimate correspondences or compatibilities at multiple representational levels rather than through a single global embedding. Earlier cross-media retrieval work introduced global, local, and relation alignments in a unified framework [1804.09539], while semantic alignment work employed pixel-wise dynamic fusion between local and contextual correlation volumes [1909.03444]. In that sense, DMWAN adopts the same general methodological direction—multi-level alignment with input-dependent weighting—but instantiates it for sketch-image retrieval under zero-shot transfer [2511.00925].

A common misunderstanding is to treat DMWAN as a generic multimodal fusion network. The reported method is more specific: it is a retrieval-oriented metric learning system in which the “alignment” signal is not itself the final retrieval score, but a dynamic scalar weight that regulates the contribution of each training pair to the loss. This suggests that the network’s primary novelty lies in training-time sample reweighting informed by cross-modal agreement, rather than in replacing standard retrieval similarity computation with a new matching operator.

## 2. System architecture

The reported DMWAN consists of three sequential modules: (i) a Uni-modal Feature Extraction Module, (ii) a Cross-modal Multi-level Weighting Module, and (iii) a Weighted Quadruplet Loss Module [2511.00925].

### Uni-modal feature extraction

For sketches and images, the backbone is ViT-B/16. The input is an RGB sketch $S\in\mathbb R^{h\times w\times 3}$ or image $I\in\mathbb R^{h\times w\times 3}$, resized to $224\times 224$. The input is split into $16\times 16$ patches, linearly projected to token embeddings, and augmented with a learnable global token $[\mathrm{Glo}]$. After $L$ layers of multi-head self-attention and MLP blocks, the model yields a global token and local patch tokens:
$$
Z^{X}=\Bigl[z_{L}^{X,\mathrm{Glo}},\,z_{L}^{X,1},\,\dots,\,z_{L}^{X,n}\Bigr],\quad X\in\{S,I\},
$$
with $n=14^2=196$ patches. The notation distinguishes
$$
X_{\mathrm{global}}=z_{L}^{X,\mathrm{Glo}}\in\mathbb R^d
$$
and
$$
X_{\mathrm{local}}=[z_{L}^{X,1},\dots,z_{L}^{X,n}]\in\mathbb R^{n\times d}.
$$
This decomposition makes the later weighting module explicitly bi-level: one pathway evaluates local alignment and the other global alignment [2511.00925].

For text, the model uses the CLIP ViT-B/16 text encoder. Prompt-based templates such as “a photo of a $c$” are applied to the category name, yielding textual tokens $T_{\mathrm{CLIP}}\in\mathbb R^{m\times d_t}$ with $d_t=512$. A learnable linear projection $f_t:\mathbb R^{d_t}\to\mathbb R^d$ maps these tokens into the same latent dimension as the visual backbones:
$$
T=f_t(T_{\mathrm{CLIP}})\in\mathbb R^{m\times d}.
$$
The text branch therefore acts as a semantic bridge between sketches and images during training [2511.00925].

### Module-level organization

The architecture can be summarized as follows.

| Component | Function | Reported elements |
|---|---|---|
| Uni-modal Feature Extraction Module | Produce local and global tokens | ViT-B/16 for sketch/image, CLIP text encoder |
| Cross-modal Multi-level Weighting Module | Estimate alignment quality | Cross-attention, Local Aggregation Block, Global Aggregation Block |
| Weighted Quadruplet Loss Module | Reweight metric learning | Weighted sum of two margin-based terms |

The architecture is “dynamic” because the alignment weights are recomputed for every mini-batch, and “multi-level” because the weights depend on both patch-level and global-token-level agreement [2511.00925].

## 3. Cross-modal multi-level weighting

The central mechanism in DMWAN is the computation of a scalar alignment weight $w_i\in(0,1]$ for each sketch-image training pair. This weight is obtained by combining local and global cross-modal assessments [2511.00925].

### Cross-attention preprocessing

Before weighting, the model applies a single layer of cross-attention with 12 heads in both directions, taking textual or sketch tokens as queries over image or local tokens as keys and values. The reported form is
$$
C\text{-Atten}_{X\leftarrow Y}(Q_Y,K_X,V_X)
=
\mathrm{Softmax}\!\bigl(Q_YK_X^T/\sqrt d\bigr)V_X.
$$
This preprocessing causes sketch or image local tokens to incorporate cross-modal context prior to measuring alignment quality [2511.00925].

### Local Aggregation Block

Let $\widetilde S_{\mathrm{local}}$ and $\widetilde I_{\mathrm{local}}\in\mathbb R^{n\times d}$ denote the cross-attended local tokens of sketch and image. For each modality, the method flattens across $d$ and applies a row-wise softmax to obtain discrete distributions over the $n$ patch positions:
$$
D_s=\mathrm{Softmax}\bigl(\widetilde S_{\mathrm{local}}\mathbf 1_d\bigr),\quad
D_i=\mathrm{Softmax}\bigl(\widetilde I_{\mathrm{local}}\mathbf 1_d\bigr),\quad \in\Delta^n.
$$
The local discrepancy is then measured by
$$
\delta_{\mathrm{KL}}^{\mathrm{local}}=\mathrm{KL}(D_s\Vert D_i).
$$
Across a batch of $B$ pairs, the resulting values are normalized and thresholded at $t_\ell$ to form the local weight list:
$$
w_{\mathrm{local}}(i)=
\begin{cases}
1, & L_{\mathrm{raw}}(i)\le t_\ell,\\
\exp\bigl(L_{\mathrm{raw}}(i)\bigr)-1, & L_{\mathrm{raw}}(i)>t_\ell.
\end{cases}
$$
The role of the Local Aggregation Block is therefore to convert patch-level agreement into a per-sample weighting factor [2511.00925].

### Global Aggregation Block

The Global Aggregation Block uses the sketch global token $S_{\mathrm{global}}$, image global token $I_{\mathrm{global}}$, and text global token $T_{\mathrm{global}}$. It constructs batch-wise cosine similarity matrices
$$
C_s(i,j)=\cos\bigl(T_{\mathrm{global}}^j,S_{\mathrm{global}}^i\bigr),\quad
C_i(i,j)=\cos\bigl(T_{\mathrm{global}}^j,I_{\mathrm{global}}^i\bigr).
$$
After row-wise softmax and summation across columns, the KL divergence between the two resulting vectors is used as the global discrepancy:
$$
\delta_{\mathrm{KL}}^{\mathrm{global}}
=
\mathrm{KL}\Bigl(\mathrm{sum}(\mathrm{Softmax}(C_i))\Vert \mathrm{sum}(\mathrm{Softmax}(C_s))\Bigr).
$$
Thresholding at $t_g$ yields
$$
w_{\mathrm{global}}(i)=
\begin{cases}
1, & G_{\mathrm{raw}}(i)\le t_g,\\
\exp\bigl(G_{\mathrm{raw}}(i)\bigr)-1, & G_{\mathrm{raw}}(i)>t_g.
\end{cases}
$$
The global branch thus estimates whether sketch and image exhibit similar category-level compatibility with the textual semantic anchor across the batch [2511.00925].

### Final alignment weight

The final sample weight is the product of the local and global terms:
$$
w_i=w_{\mathrm{local}}(i)\times w_{\mathrm{global}}(i),\quad i=1,\dots,B.
$$
This multiplicative composition means that both local and global agreement influence the final weight. A plausible implication is that a pair must be reasonably consistent at both representational levels to receive a strong contribution to the metric objective.

## 4. Weighted quadruplet objective and optimization

The Weighted Quadruplet Loss Module is designed to balance modalities and to de-emphasize poorly aligned pairs [2511.00925]. For each anchor sketch $s_{\mathrm{anc}}$, the method forms a positive image $i_{\mathrm{pos}}$ of the same class, a negative image $i_{\mathrm{neg}}$ of a different class, and a negative sketch $s_{\mathrm{neg}}$ of a different class.

Two margin-based terms are defined:
$$
L_o(i)=
\Bigl[
\lVert s_{\mathrm{anc}}^i-i_{\mathrm{pos}}^i\rVert_2
-
\lVert s_{\mathrm{anc}}^i-i_{\mathrm{neg}}^i\rVert_2
+\alpha
\Bigr]_+,
$$
and
$$
L_d(i)=
\Bigl[
\lVert s_{\mathrm{anc}}^i-i_{\mathrm{pos}}^i\rVert_2
-
\lVert s_{\mathrm{anc}}^i-s_{\mathrm{neg}}^i\rVert_2
+\beta
\Bigr]_+,
$$
where $[x]_+=\max\{x,0\}$ and $\alpha,\beta>0$ are hyper-margins. The final batch loss is
$$
L_{\mathrm{WQuad}}
=
\frac1B\sum_{i=1}^B w_i\bigl(L_o(i)+L_d(i)\bigr).
$$
The reported implementation uses $\alpha=0.3$ and $\beta=0.2$, with Adam, learning rate $5\times 10^{-6}$, weight decay $10^{-2}$, and batch size $64$ [2511.00925].

The training procedure follows a fixed order. First, the model computes sketch/image local and global tokens and obtains projected text tokens. Second, it computes local and global alignment weights by cross-attention, KL-based discrepancy estimation, and thresholding. Third, it constructs domain-balanced quadruplets from the mini-batch. Fourth, it evaluates the weighted quadruplet loss and updates the network parameters [2511.00925].

An important operational point is that all alignment weights $w_i$ are recomputed each mini-batch. This is the direct source of the model’s “dynamic” behavior. During inference, by contrast, only sketch and image features and their cross-attention are used for retrieval scoring; text is not used at test time [2511.00925]. This clarifies that the text encoder is a training-time scaffold rather than a required retrieval-time modality.

## 5. Empirical results and ablations

The reported evaluation covers three benchmark datasets: Sketchy, TU-Berlin, and QuickDraw. The protocol includes Sketchy-Ext with a Sketchy-25 split of 100 seen and 25 unseen classes, a Sketchy-21 split of 104 seen and 21 non-ImageNet unseen classes, TU-Berlin with 220 seen and 30 unseen classes, and QuickDraw with 80 seen and 30 unseen classes. The metrics are mean average precision (mAP@all) and Precision@K with $K=100$ or $200$ [2511.00925].

### Quantitative comparison

The paper reports the following mAP@all values:

| Dataset | DMWAN | Best prior cited in the details |
|---|---:|---:|
| TU-Berlin | 0.623 | 0.617 |
| Sketchy-25 | 0.787 | 0.763 |
| Sketchy-21 | 0.572 | 0.532 |
| QuickDraw | 0.170 | 0.161 |

These values are presented as improvements over CMAAN-ATD on TU-Berlin, Sketchy-25, and QuickDraw, and over ZS-SBIR-Adaptive on Sketchy-21 [2511.00925].

### Ablation structure

The ablation study on Sketchy-25 and Sketchy-21 reports:

| Variant | Sketchy-25 | Sketchy-21 |
|---|---:|---:|
| base (no weighting, standard triplet) | 0.763 | 0.515 |
| + local weight only (triplet) | 0.770 | 0.538 |
| + local weight + weighted quadruplet | 0.786 | 0.566 |
| full (local + global weighting + quadruplet) | 0.787 | 0.572 |

These results isolate three effects. First, local weighting alone improves over the base model. Second, replacing the standard triplet with the weighted quadruplet gives a larger improvement. Third, adding the global weighting term on top of the local weighting and weighted quadruplet produces the best result [2511.00925].

A plausible interpretation is that the local branch captures fine-grained sketch-image consistency, while the global branch adds class-level semantic stabilization via the text anchor. The gain from the weighted quadruplet further suggests that alignment quality estimation and domain balancing are complementary rather than redundant.

## 6. Relation to adjacent alignment models

DMWAN should be situated among several neighboring strands of research on multi-level alignment, though its task and mechanics are distinct [2511.00925].

In cross-media retrieval, CRAN learns global, local, and relation alignments and combines their similarity scores with equal weight at inference, while its only dynamic weights are attention vectors recomputed for each sample [1804.09539]. In semantic correspondence, DCCNet fuses local and context correlation volumes through a pixel-wise softmax attention mask and supplements the fused branch with auxiliary losses [1909.03444]. In real-time segmentation, MFARANet aligns multi-scale score maps recursively and fuses them through an attention-based Adaptive Scores Fusion Module that assigns per-pixel weights to aligned predictions [2402.02286]. These systems share the broad principle that representational levels should not contribute uniformly for every input.

DMWAN differs from those models in two respects. First, its dynamic weighting is attached to training pairs rather than to retrieval-time similarity aggregation or dense spatial fusion. Second, its multi-level structure is explicitly divided into a local patch-level KL comparison and a global text-mediated KL comparison, followed by multiplicative fusion into a single scalar loss weight [2511.00925].

A second misconception is to conflate DMWAN with generic “dynamic weighted alignment” approaches outside retrieval. The same expression can describe preference-weighted language model alignment [2506.19780] or adaptive multi-layer fusion in segmentation-style systems, but in DMWAN the term refers specifically to cross-modal alignment quality estimation for sketch-image pairs in ZS-SBIR [2511.00925]. This suggests that the name should be interpreted in the context of its retrieval formulation rather than as a universal architectural template.

## 7. Technical interpretation and significance

The reported design encodes a particular view of noise and modality imbalance in ZS-SBIR. Instead of assuming that every sketch-image pair in a mini-batch is equally informative, the model estimates how well the pair aligns at two levels and uses that estimate to modulate the metric loss [2511.00925]. This gives the training objective a data-dependent anisotropy: harder or less consistent examples need not influence the parameter update in the same way as better-aligned examples.

The use of text is also structurally significant. At the local level, cross-attention injects cross-modal context into visual tokens. At the global level, text functions as a shared semantic reference for comparing sketch and image compatibility across the batch [2511.00925]. Because text is removed at test time, the method can be read as a training-time semantic regularization strategy for learning improved sketch and image representations under zero-shot conditions.

From a broader perspective, DMWAN exemplifies a recurrent pattern in alignment research: multi-level descriptors are combined with dynamic weighting to compensate for the limitations of single-level matching. In CRAN this appears as global/local/relation alignment [1804.09539]; in DCCNet as local/context fusion [1909.03444]; and in MFARANet as scale-adaptive score fusion after recursive alignment [2402.02286]. DMWAN’s contribution within this pattern is to transpose dynamic multi-level weighting into ZS-SBIR and couple it directly to a weighted quadruplet objective [2511.00925].

Source: https://www.emergentmind.com/topics/dynamic-multi-level-weighted-alignment-network