Dynamic Multi-level Weighted Alignment Network

Updated 5 July 2026

The paper demonstrates that dynamic weighting of sketch-image pairs mitigates imbalanced modalities and poor alignment during training.
It decomposes the process into a uni-modal extraction using ViT and a cross-modal aggregation incorporating both local patch and global text-guided features.
Empirical results on Sketchy, TU-Berlin, and QuickDraw show superior retrieval performance with ablation studies validating the impact of each module.

Searching arXiv for the primary paper and closely related alignment works to ground the article in cited sources. Dynamic Multi-level Weighted Alignment Network (DMWAN) is a zero-shot sketch-based image retrieval (ZS-SBIR) architecture that addresses two training-time deficiencies identified in prior work: the use of imbalanced samples of modalities and the presence of inconsistent low-quality information, both of which lead to sub-optimal performance. The model comprises three components: a Uni-modal Feature Extraction Module, a Cross-modal Multi-level Weighting Module, and a Weighted Quadruplet Loss Module. In the reported formulation, sketch and image inputs are encoded by a ViT, category text is encoded by a CLIP text encoder, alignment quality is estimated from both local and global cross-modal agreement, and the resulting dynamic weights modulate a quadruplet-style metric learning objective. Experiments on Sketchy, TU-Berlin, and QuickDraw are reported to show superior performance over state-of-the-art ZS-SBIR methods (Su et al., 2 Nov 2025).

1. Problem formulation and scope

DMWAN is proposed for ZS-SBIR, where a sketch query must retrieve images from unseen classes at test time. The motivating claim is that previous methods suffer from two related issues during training: imbalanced samples of modalities and inconsistent low-quality information. The network is designed to mitigate both by assigning a sample-dependent alignment weight to each sketch-image pair and by incorporating that weight into a domain-balanced metric loss (Su et al., 2 Nov 2025).

Within the broader alignment literature, DMWAN belongs to a family of methods that estimate correspondences or compatibilities at multiple representational levels rather than through a single global embedding. Earlier cross-media retrieval work introduced global, local, and relation alignments in a unified framework (Qi et al., 2018), while semantic alignment work employed pixel-wise dynamic fusion between local and contextual correlation volumes (Huang et al., 2019). In that sense, DMWAN adopts the same general methodological direction—multi-level alignment with input-dependent weighting—but instantiates it for sketch-image retrieval under zero-shot transfer (Su et al., 2 Nov 2025).

A common misunderstanding is to treat DMWAN as a generic multimodal fusion network. The reported method is more specific: it is a retrieval-oriented metric learning system in which the “alignment” signal is not itself the final retrieval score, but a dynamic scalar weight that regulates the contribution of each training pair to the loss. This suggests that the network’s primary novelty lies in training-time sample reweighting informed by cross-modal agreement, rather than in replacing standard retrieval similarity computation with a new matching operator.

2. System architecture

The reported DMWAN consists of three sequential modules: (i) a Uni-modal Feature Extraction Module, (ii) a Cross-modal Multi-level Weighting Module, and (iii) a Weighted Quadruplet Loss Module (Su et al., 2 Nov 2025).

For sketches and images, the backbone is ViT-B/16. The input is an RGB sketch $S\in\mathbb R^{h\times w\times 3}$ or image $I\in\mathbb R^{h\times w\times 3}$ , resized to $224\times 224$ . The input is split into $16\times 16$ patches, linearly projected to token embeddings, and augmented with a learnable global token $[\mathrm{Glo}]$ . After $L$ layers of multi-head self-attention and MLP blocks, the model yields a global token and local patch tokens:

$Z^{X}=\Bigl[z_{L}^{X,\mathrm{Glo}},\,z_{L}^{X,1},\,\dots,\,z_{L}^{X,n}\Bigr],\quad X\in\{S,I\},$

with $n=14^2=196$ patches. The notation distinguishes

$X_{\mathrm{global}}=z_{L}^{X,\mathrm{Glo}}\in\mathbb R^d$

and

$X_{\mathrm{local}}=[z_{L}^{X,1},\dots,z_{L}^{X,n}]\in\mathbb R^{n\times d}.$

This decomposition makes the later weighting module explicitly bi-level: one pathway evaluates local alignment and the other global alignment (Su et al., 2 Nov 2025).

For text, the model uses the CLIP ViT-B/16 text encoder. Prompt-based templates such as “a photo of a $I\in\mathbb R^{h\times w\times 3}$ 0” are applied to the category name, yielding textual tokens $I\in\mathbb R^{h\times w\times 3}$ 1 with $I\in\mathbb R^{h\times w\times 3}$ 2. A learnable linear projection $I\in\mathbb R^{h\times w\times 3}$ 3 maps these tokens into the same latent dimension as the visual backbones:

$I\in\mathbb R^{h\times w\times 3}$ 4

The text branch therefore acts as a semantic bridge between sketches and images during training (Su et al., 2 Nov 2025).

Module-level organization

The architecture can be summarized as follows.

Component	Function	Reported elements
Uni-modal Feature Extraction Module	Produce local and global tokens	ViT-B/16 for sketch/image, CLIP text encoder
Cross-modal Multi-level Weighting Module	Estimate alignment quality	Cross-attention, Local Aggregation Block, Global Aggregation Block
Weighted Quadruplet Loss Module	Reweight metric learning	Weighted sum of two margin-based terms

The architecture is “dynamic” because the alignment weights are recomputed for every mini-batch, and “multi-level” because the weights depend on both patch-level and global-token-level agreement (Su et al., 2 Nov 2025).

The central mechanism in DMWAN is the computation of a scalar alignment weight $I\in\mathbb R^{h\times w\times 3}$ 5 for each sketch-image training pair. This weight is obtained by combining local and global cross-modal assessments (Su et al., 2 Nov 2025).

Cross-attention preprocessing

Before weighting, the model applies a single layer of cross-attention with 12 heads in both directions, taking textual or sketch tokens as queries over image or local tokens as keys and values. The reported form is

$I\in\mathbb R^{h\times w\times 3}$ 6

This preprocessing causes sketch or image local tokens to incorporate cross-modal context prior to measuring alignment quality (Su et al., 2 Nov 2025).

Local Aggregation Block

Let $I\in\mathbb R^{h\times w\times 3}$ 7 and $I\in\mathbb R^{h\times w\times 3}$ 8 denote the cross-attended local tokens of sketch and image. For each modality, the method flattens across $I\in\mathbb R^{h\times w\times 3}$ 9 and applies a row-wise softmax to obtain discrete distributions over the $224\times 224$ 0 patch positions:

$224\times 224$ 1

The local discrepancy is then measured by

$224\times 224$ 2

Across a batch of $224\times 224$ 3 pairs, the resulting values are normalized and thresholded at $224\times 224$ 4 to form the local weight list:

$224\times 224$ 5

The role of the Local Aggregation Block is therefore to convert patch-level agreement into a per-sample weighting factor (Su et al., 2 Nov 2025).

Global Aggregation Block

The Global Aggregation Block uses the sketch global token $224\times 224$ 6, image global token $224\times 224$ 7, and text global token $224\times 224$ 8. It constructs batch-wise cosine similarity matrices

$224\times 224$ 9

After row-wise softmax and summation across columns, the KL divergence between the two resulting vectors is used as the global discrepancy:

$16\times 16$ 0

Thresholding at $16\times 16$ 1 yields

$16\times 16$ 2

The global branch thus estimates whether sketch and image exhibit similar category-level compatibility with the textual semantic anchor across the batch (Su et al., 2 Nov 2025).

Final alignment weight

The final sample weight is the product of the local and global terms:

$16\times 16$ 3

This multiplicative composition means that both local and global agreement influence the final weight. A plausible implication is that a pair must be reasonably consistent at both representational levels to receive a strong contribution to the metric objective.

4. Weighted quadruplet objective and optimization

The Weighted Quadruplet Loss Module is designed to balance modalities and to de-emphasize poorly aligned pairs (Su et al., 2 Nov 2025). For each anchor sketch $16\times 16$ 4, the method forms a positive image $16\times 16$ 5 of the same class, a negative image $16\times 16$ 6 of a different class, and a negative sketch $16\times 16$ 7 of a different class.

Two margin-based terms are defined:

$16\times 16$ 8

and

$16\times 16$ 9

where $[\mathrm{Glo}]$ 0 and $[\mathrm{Glo}]$ 1 are hyper-margins. The final batch loss is

$[\mathrm{Glo}]$ 2

The reported implementation uses $[\mathrm{Glo}]$ 3 and $[\mathrm{Glo}]$ 4, with Adam, learning rate $[\mathrm{Glo}]$ 5, weight decay $[\mathrm{Glo}]$ 6, and batch size $[\mathrm{Glo}]$ 7 (Su et al., 2 Nov 2025).

The training procedure follows a fixed order. First, the model computes sketch/image local and global tokens and obtains projected text tokens. Second, it computes local and global alignment weights by cross-attention, KL-based discrepancy estimation, and thresholding. Third, it constructs domain-balanced quadruplets from the mini-batch. Fourth, it evaluates the weighted quadruplet loss and updates the network parameters (Su et al., 2 Nov 2025).

An important operational point is that all alignment weights $[\mathrm{Glo}]$ 8 are recomputed each mini-batch. This is the direct source of the model’s “dynamic” behavior. During inference, by contrast, only sketch and image features and their cross-attention are used for retrieval scoring; text is not used at test time (Su et al., 2 Nov 2025). This clarifies that the text encoder is a training-time scaffold rather than a required retrieval-time modality.

5. Empirical results and ablations

The reported evaluation covers three benchmark datasets: Sketchy, TU-Berlin, and QuickDraw. The protocol includes Sketchy-Ext with a Sketchy-25 split of 100 seen and 25 unseen classes, a Sketchy-21 split of 104 seen and 21 non-ImageNet unseen classes, TU-Berlin with 220 seen and 30 unseen classes, and QuickDraw with 80 seen and 30 unseen classes. The metrics are mean average precision (mAP@all) and Precision@K with $[\mathrm{Glo}]$ 9 or $L$ 0 (Su et al., 2 Nov 2025).

Quantitative comparison

The paper reports the following mAP@all values:

Dataset	DMWAN	Best prior cited in the details
TU-Berlin	0.623	0.617
Sketchy-25	0.787	0.763
Sketchy-21	0.572	0.532
QuickDraw	0.170	0.161

These values are presented as improvements over CMAAN-ATD on TU-Berlin, Sketchy-25, and QuickDraw, and over ZS-SBIR-Adaptive on Sketchy-21 (Su et al., 2 Nov 2025).

Ablation structure

The ablation study on Sketchy-25 and Sketchy-21 reports:

Variant	Sketchy-25	Sketchy-21
base (no weighting, standard triplet)	0.763	0.515
+ local weight only (triplet)	0.770	0.538
+ local weight + weighted quadruplet	0.786	0.566
full (local + global weighting + quadruplet)	0.787	0.572

These results isolate three effects. First, local weighting alone improves over the base model. Second, replacing the standard triplet with the weighted quadruplet gives a larger improvement. Third, adding the global weighting term on top of the local weighting and weighted quadruplet produces the best result (Su et al., 2 Nov 2025).

A plausible interpretation is that the local branch captures fine-grained sketch-image consistency, while the global branch adds class-level semantic stabilization via the text anchor. The gain from the weighted quadruplet further suggests that alignment quality estimation and domain balancing are complementary rather than redundant.

6. Relation to adjacent alignment models

DMWAN should be situated among several neighboring strands of research on multi-level alignment, though its task and mechanics are distinct (Su et al., 2 Nov 2025).

In cross-media retrieval, CRAN learns global, local, and relation alignments and combines their similarity scores with equal weight at inference, while its only dynamic weights are attention vectors recomputed for each sample (Qi et al., 2018). In semantic correspondence, DCCNet fuses local and context correlation volumes through a pixel-wise softmax attention mask and supplements the fused branch with auxiliary losses (Huang et al., 2019). In real-time segmentation, MFARANet aligns multi-scale score maps recursively and fuses them through an attention-based Adaptive Scores Fusion Module that assigns per-pixel weights to aligned predictions (Zhang et al., 2024). These systems share the broad principle that representational levels should not contribute uniformly for every input.

DMWAN differs from those models in two respects. First, its dynamic weighting is attached to training pairs rather than to retrieval-time similarity aggregation or dense spatial fusion. Second, its multi-level structure is explicitly divided into a local patch-level KL comparison and a global text-mediated KL comparison, followed by multiplicative fusion into a single scalar loss weight (Su et al., 2 Nov 2025).

A second misconception is to conflate DMWAN with generic “dynamic weighted alignment” approaches outside retrieval. The same expression can describe preference-weighted LLM alignment (Sun et al., 24 Jun 2025) or adaptive multi-layer fusion in segmentation-style systems, but in DMWAN the term refers specifically to cross-modal alignment quality estimation for sketch-image pairs in ZS-SBIR (Su et al., 2 Nov 2025). This suggests that the name should be interpreted in the context of its retrieval formulation rather than as a universal architectural template.

7. Technical interpretation and significance

The reported design encodes a particular view of noise and modality imbalance in ZS-SBIR. Instead of assuming that every sketch-image pair in a mini-batch is equally informative, the model estimates how well the pair aligns at two levels and uses that estimate to modulate the metric loss (Su et al., 2 Nov 2025). This gives the training objective a data-dependent anisotropy: harder or less consistent examples need not influence the parameter update in the same way as better-aligned examples.

The use of text is also structurally significant. At the local level, cross-attention injects cross-modal context into visual tokens. At the global level, text functions as a shared semantic reference for comparing sketch and image compatibility across the batch (Su et al., 2 Nov 2025). Because text is removed at test time, the method can be read as a training-time semantic regularization strategy for learning improved sketch and image representations under zero-shot conditions.

From a broader perspective, DMWAN exemplifies a recurrent pattern in alignment research: multi-level descriptors are combined with dynamic weighting to compensate for the limitations of single-level matching. In CRAN this appears as global/local/relation alignment (Qi et al., 2018); in DCCNet as local/context fusion (Huang et al., 2019); and in MFARANet as scale-adaptive score fusion after recursive alignment (Zhang et al., 2024). DMWAN’s contribution within this pattern is to transpose dynamic multi-level weighting into ZS-SBIR and couple it directly to a weighted quadruplet objective (Su et al., 2 Nov 2025).