Windowed Cross-Attention Module

• Windowed cross-attention is an attention mechanism where queries attend to keys and values from a different source within constrained local windows.
• It is applied in domains like medical image registration and asynchronous speech enhancement to reduce computational cost while aligning features.
• The approach focuses on defining local windows, managing fixed versus deformable sampling, and balancing efficiency with effective cross-modal interaction.

Windowed cross-attention denotes a class of attention mechanisms in which queries from one source attend to keys and values from another source only within constrained local neighborhoods rather than over the full token set. In the literature considered here, the locality constraint appears as fixed temporal windows, fixed or deformable spatial windows, and subsequence-restricted bands. The term is used most literally for deformable cross-attention in medical image registration and for temporal alignment across asynchronous microphones in speech enhancement (Chen et al., 2023, Yang et al., 21 Jul 2025). Related work in retrieval adopts an asymmetric subsequence formulation that combines local document windows with query-aware cross-interaction (Schlatt et al., 2023). By contrast, several vision modules often discussed alongside windowed cross-attention are explicitly windowed self-attention, cross-window communication, or non-attention mixing mechanisms rather than standard cross-attention in the query-key-value sense (Shang et al., 2023, Mian et al., 25 Feb 2025).

1. Terminological scope and distinguishing criteria

The most stable distinction in the literature is between windowed cross-attention, windowed self-attention, and cross-window communication. A module is a literal windowed cross-attention mechanism when queries and keys-values come from different sources and interaction is restricted to a local window. A module is only analogous when it solves cross-window information exchange through windowed self-attention, structured sparsity, spectral mixing, or global pooled attention.

Work	Interaction type	Locality mechanism
TM-DCA (Chen et al., 2023)	Cross-attention between moving and fixed images	Regular base windows with deformable reference sampling
Distributed speech WCA (Yang et al., 21 Jul 2025)	Cross-attention across microphones	Temporal window $i-L \le j \le i+L$
Sparse cross-encoder (Schlatt et al., 2023)	Asymmetric cross-attention between subsequences	Document-side local window $w$
MSCSA (Shang et al., 2023)	Not standard windowed cross-attention	Pooled multi-scale global attention
FwNet-ECA (Mian et al., 25 Feb 2025)	Not cross-attention in the standard query-key-value sense	Windowed self-attention plus Fourier Filter Enhancement

This distinction matters because several influential vision architectures target the same systems problem—reducing the cost of all-pairs interaction while enlarging receptive field—without instantiating cross-attention. CSWin uses cross-shaped window self-attention over horizontal and vertical stripes (Dong et al., 2021). AEWin combines local window attention with horizontal and vertical axial attention (Zhang et al., 2022). Swin-Free removes shifted windows and uses size-varying windows across stages to create cross-window interaction (Koo et al., 2023). Gated-SwinRMT keeps Swin’s shifted-window scaffold but replaces its internal kernel with decomposed Manhattan self-attention and gating (Maity et al., 7 Apr 2026). Iwin uses interleaved window self-attention with depthwise separable convolution (Huo et al., 24 Jul 2025). These mechanisms are adjacent to the topic, but they are not interchangeable with literal windowed cross-attention.

A recurrent misconception is to equate any efficient local attention block with a windowed cross-attention module. The surveyed papers explicitly separate these categories. MSCSA is described as a global attention over a pooled multi-scale token set and “does not partition the feature map into windows” for attention computation (Shang et al., 2023). FwNet-ECA positions its Filter Enhancement layer as a cross-window communication mechanism, but “it is not attention in the standard query-key-value sense” (Mian et al., 25 Feb 2025).

2. Core mathematical template

The generic cross-attention template remains the standard scaled dot-product operator, but windowing constrains which query-key pairs are instantiated. In the registration formulation, the base tokens produce keys and values and the reference tokens produce queries: $$[\mathbf{K}_b,\mathbf{V}_b] = \mathbf{x}_b \mathbf{U}_{k,v}, \qquad \mathbf{Q}_r = \mathbf{x}_r \mathbf{U}_q,$$

$$CA(\mathbf{x}) = \mathrm{softmax}\!\left(\frac{\mathbf{Q}_r \mathbf{K}_b^\top}{\sqrt{D_k}}\right)\mathbf{V}_b.$$

Windowing does not change this attention core; it changes how keys, values, and queries are partitioned or sampled (Chen et al., 2023).

Three design axes recur across the literature. The first is how the window is defined. In asynchronous speech enhancement, frame $i$ in microphone $m$ attends only to frames $j$ in microphone $n$ satisfying

$i-L \leq j \leq i+L,$

and the keys and values are unfolded into

$$K_n^u, V_n^u \in \mathbb{R}^{T \times (2L+1) \times d}.$$

The resulting local cross-attention is

$w$0

This makes the locality prior explicit in time rather than in image space (Yang et al., 21 Jul 2025).

The second axis is whether the window is fixed or deformable. TM-DCA predicts offsets

$w$1

and then samples the reference image at displaced coordinates,

$w$2

so that the effective search region can be much larger than the computational window (Chen et al., 2023).

The third axis is whether cross-attention is symmetric across subsequences. In the sparse cross-encoder formulation, the input is split into [CLS], query tokens, and passage/document tokens. The final attention pattern is

$w$3

$w$4

$w$5

Here [CLS] attends globally, query tokens attend only within the query, and document tokens attend to [CLS], query tokens, and a local document window of size $w$6 (Schlatt et al., 2023).

Across these formulations, the window is not merely a sparsity mask. It encodes an application-specific prior: bounded temporal delay, bounded spatial correspondence, or bounded local discourse continuity. A plausible implication is that the utility of a windowed cross-attention module depends less on the existence of locality per se than on whether the imposed neighborhood matches the task’s underlying alignment structure.

3. Deformable spatial correspondence in medical image registration

The Deformable Cross-Attention Transformer introduces a windowed cross-attention module for medical image registration in which the model must identify correspondences between a moving image and a fixed image under potentially large anatomical displacement (Chen et al., 2023). The paper frames the design against two baselines. Global cross-attention, as in Attention-Reg, is broad but applied only at a single low resolution, so it does not support the hierarchical feature fusion used in registration models. Fixed local cross-attention with expanded windows, as in XMorpher, enlarges the search window from

$w$7

to

$w$8

with $w$9, increasing complexity from

$$[\mathbf{K}_b,\mathbf{V}_b] = \mathbf{x}_b \mathbf{U}_{k,v}, \qquad \mathbf{Q}_r = \mathbf{x}_r \mathbf{U}_q,$$0

to

$$[\mathbf{K}_b,\mathbf{V}_b] = \mathbf{x}_b \mathbf{U}_{k,v}, \qquad \mathbf{Q}_r = \mathbf{x}_r \mathbf{U}_q,$$1

With $$[\mathbf{K}_b,\mathbf{V}_b] = \mathbf{x}_b \mathbf{U}_{k,v}, \qquad \mathbf{Q}_r = \mathbf{x}_r \mathbf{U}_q,$$2, the attention cost increases by roughly $$[\mathbf{K}_b,\mathbf{V}_b] = \mathbf{x}_b \mathbf{U}_{k,v}, \qquad \mathbf{Q}_r = \mathbf{x}_r \mathbf{U}_q,$$3.

TM-DCA keeps the attention window size fixed and instead learns where to sample inside the reference image. The base image is partitioned into regular windows to produce keys and values: $$[\mathbf{K}_b,\mathbf{V}_b] = \mathbf{x}_b \mathbf{U}_{k,v}, \qquad \mathbf{Q}_r = \mathbf{x}_r \mathbf{U}_q,$$4 A lightweight two-layer convolutional offset network, consisting of a depth-wise convolution followed by a regular convolution, predicts offsets from the added moving and fixed features. Those offsets deform the reference sampling grid, and the sampled reference features are projected into queries. Sampling uses trilinear interpolation. The resulting module preserves fixed-size windowed attention while allowing sampling beyond the original window, potentially over the whole image.

The paper emphasizes that this decouples search range from attention size. Tokens can be sampled beyond the predefined local window, windows can overlap, and the cross-attention computation remains fixed-size. The overall complexity is stated as

$$[\mathbf{K}_b,\mathbf{V}_b] = \mathbf{x}_b \mathbf{U}_{k,v}, \qquad \mathbf{Q}_r = \mathbf{x}_r \mathbf{U}_q,$$5

far below XMorpher’s

$$[\mathbf{K}_b,\mathbf{V}_b] = \mathbf{x}_b \mathbf{U}_{k,v}, \qquad \mathbf{Q}_r = \mathbf{x}_r \mathbf{U}_q,$$6

This formulation therefore uses deformation to recover large effective receptive fields without geometrically enlarging the attention tensor itself.

Architecturally, the module is embedded in a dual-encoder Transformer registration model with parallel Transformer encoders for moving and fixed images. At each resolution, $$[\mathbf{K}_b,\mathbf{V}_b] = \mathbf{x}_b \mathbf{U}_{k,v}, \qquad \mathbf{Q}_r = \mathbf{x}_r \mathbf{U}_q,$$7 successive deformable cross-attention modules fuse the two streams. The reported encoder layer counts are $$[\mathbf{K}_b,\mathbf{V}_b] = \mathbf{x}_b \mathbf{U}_{k,v}, \qquad \mathbf{Q}_r = \mathbf{x}_r \mathbf{U}_q,$$8, the window sizes are $$[\mathbf{K}_b,\mathbf{V}_b] = \mathbf{x}_b \mathbf{U}_{k,v}, \qquad \mathbf{Q}_r = \mathbf{x}_r \mathbf{U}_q,$$9, the embedding dimension is 96, and successive blocks use shifted window partitioning to introduce cross-window connections. A ConvNet decoder is retained for deformation refinement.

Experimentally, TM-DCA outperformed the compared methods on all three reported registration tasks: OASIS mono-modal registration with Dice $$CA(\mathbf{x}) = \mathrm{softmax}\!\left(\frac{\mathbf{Q}_r \mathbf{K}_b^\top}{\sqrt{D_k}}\right)\mathbf{V}_b.$$0, IXI atlas-to-patient registration with Dice $$CA(\mathbf{x}) = \mathrm{softmax}\!\left(\frac{\mathbf{Q}_r \mathbf{K}_b^\top}{\sqrt{D_k}}\right)\mathbf{V}_b.$$1, and ALBERTs multi-modal registration with Dice $$CA(\mathbf{x}) = \mathrm{softmax}\!\left(\frac{\mathbf{Q}_r \mathbf{K}_b^\top}{\sqrt{D_k}}\right)\mathbf{V}_b.$$2. The paper also reports that the original XMorpher model could not fit on the GPU for the paper’s image size because memory requirements exceeded 48 GB, and therefore uses an XMorpher encoder with the TM-DCA decoder for comparison. Within the surveyed literature, this is the clearest example of a literal spatial windowed cross-attention module whose purpose is explicit correspondence estimation rather than general feature mixing.

4. Temporal alignment in asynchronous multi-microphone speech enhancement

In distributed asynchronous device speech enhancement, the central difficulty is that microphones are not time-aligned. The paper models the asynchronous observation as

$$CA(\mathbf{x}) = \mathrm{softmax}\!\left(\frac{\mathbf{Q}_r \mathbf{K}_b^\top}{\sqrt{D_k}}\right)\mathbf{V}_b.$$3

where $$CA(\mathbf{x}) = \mathrm{softmax}\!\left(\frac{\mathbf{Q}_r \mathbf{K}_b^\top}{\sqrt{D_k}}\right)\mathbf{V}_b.$$4 captures device-specific latency and $$CA(\mathbf{x}) = \mathrm{softmax}\!\left(\frac{\mathbf{Q}_r \mathbf{K}_b^\top}{\sqrt{D_k}}\right)\mathbf{V}_b.$$5 captures clock drift (Yang et al., 21 Jul 2025). Under this condition, transform-average-concatenate is insufficient because it averages hidden features only at the same time index: $$CA(\mathbf{x}) = \mathrm{softmax}\!\left(\frac{\mathbf{Q}_r \mathbf{K}_b^\top}{\sqrt{D_k}}\right)\mathbf{V}_b.$$6 which assumes synchrony that may not hold when delays are tens or even over 100 ms.

The proposed Windowed Cross-Attention module replaces same-index averaging with learned temporal alignment across microphones. It operates on hidden representations

$$CA(\mathbf{x}) = \mathrm{softmax}\!\left(\frac{\mathbf{Q}_r \mathbf{K}_b^\top}{\sqrt{D_k}}\right)\mathbf{V}_b.$$7

projects each microphone stream into

$$CA(\mathbf{x}) = \mathrm{softmax}\!\left(\frac{\mathbf{Q}_r \mathbf{K}_b^\top}{\sqrt{D_k}}\right)\mathbf{V}_b.$$8

and aggregates information for target microphone $$CA(\mathbf{x}) = \mathrm{softmax}\!\left(\frac{\mathbf{Q}_r \mathbf{K}_b^\top}{\sqrt{D_k}}\right)\mathbf{V}_b.$$9 from every microphone $i$0 through

$i$1

To avoid the $i$2 memory complexity of full pairwise alignment, attention is restricted to a temporal neighborhood. Keys and values are unfolded into local windows of width $i$3, and frame $i$4 in one microphone attends only to nearby frames in another microphone. This yields a fixed temporal search region that remains compatible with streaming.

The paper treats permutation and microphone-count invariance as primary design constraints. The same projections are applied to all streams, and each microphone aggregates information symmetrically from the set of microphones rather than from a fixed ordered tuple. After attention, the aggregated representation is fused with the original stream: $i$5 The module therefore augments rather than replaces single-microphone content.

Two backbone integrations are reported. In iFaSNet, TAC is replaced by WCA after each of 6 DPRNN blocks with hidden dimension 128. Because DPRNN outputs are chunked, the hidden states are converted from $i$6 back to sequential form $i$7 before WCA and re-segmented afterward. In CRUSE, WCA is inserted after the bottleneck of a U-Net-style model with 4 encoder and 4 decoder layers, filters $i$8, and one GRU at the bottleneck. For both systems, the STFT window length is 20 ms, the overlap is 50%, and the WCA window corresponds to $i$9, namely a 90 ms cross-attention window.

The training target is also redefined for multi-talker asynchronous scenes. Besides random microphone selection and minimum-latency selection,

$m$0

the paper proposes a “closest microphone to each speaker” target, and reports that CM consistently gives the best overall performance. Under this target, the gains over TAC are substantial. For iFaSNet, TAC OVRL is 1.94 and WCA OVRL is 2.35; TAC SIG is 2.38 and WCA SIG is 2.74. For CRUSE, TAC OVRL is 1.92 and WCA OVRL is 2.41; TAC XLSR is 2.15 and WCA XLSR is 2.35. The abstract further states that WCA offers faster convergence and improved learning. Within this application, windowed cross-attention functions as a learned local synchronization mechanism for ad-hoc microphone arrays.

5. Asymmetric subsequence attention in cross-encoders

Cross-encoders for passage and document re-ranking provide a different interpretation of windowed cross-attention. Rather than aligning two sensory streams, the model partitions a single concatenated sequence into [CLS], query tokens, and passage/document tokens, then assigns different attention privileges to each subsequence (Schlatt et al., 2023). The resulting pattern is explicitly asymmetric: query tokens do not attend to passage/document tokens, passage/document tokens can attend to query tokens, [CLS] attends to everything, and passage/document tokens use local windowed self-attention within their own subsequence.

The document-side local window is defined so that a token $m$1 attends to $m$2 tokens on the left, itself, and $m$3 tokens on the right. The paper formalizes windowed attention as

$m$4

with zero-padding when the window crosses sequence boundaries. More generally, different subsequences can receive different window sizes through

$m$5

The proposed final pattern keeps full access to [CLS], blocks query-to-document attention entirely, and windows only the document-to-document block.

This formulation is noteworthy because it rejects symmetric query-document interaction as unnecessary for effective re-ranking. The paper states that the query tokens do not need to attend to the passage or document tokens for effective re-ranking, and that very small window sizes suffice. The tested windows are

$m$6

On passages, windows 64 and 4 are statistically equivalent for the proposed sparse cross-encoder, and even $m$7 is only slightly worse. On documents, the sparse cross-encoder with $m$8 achieves average nDCG@10 of 0.594, $m$9 achieves 0.590, and $j$0 achieves 0.577.

The efficiency results are central. The paper reports that even windows of 4 tokens still yield effectiveness on par with previous cross-encoders while reducing memory requirements by at least 22% for passages and 59% for documents, and being 1% faster at inference time for passages and 43% faster for documents. In the concrete timing table, full-attention passages require 368 μs and 9 MB, while the sparse cross-encoder with $j$1 requires 364 μs and 7 MB. For documents, Longformer $j$2 requires 14 ms and 160 MB, sparse cross-encoder $j$3 requires 12 ms and 111 MB, and sparse cross-encoder $j$4 requires 8 ms and 66 MB.

The module therefore occupies an intermediate position between cross-attention and self-attention. Query tokens use self-attention only, document tokens combine windowed self-attention with cross-attention to query and [CLS], and [CLS] remains globally connected. A plausible implication is that windowed cross-attention in re-ranking is best understood as an asymmetric relevance routing pattern rather than as a symmetric two-stream interaction mechanism.

6. Relation to adjacent windowed interaction mechanisms

A large surrounding literature addresses the same efficiency-receptive-field trade-off without implementing literal windowed cross-attention. The distinction is especially important in vision. CSWin Transformer replaces square windows with cross-shaped windows formed by horizontal and vertical stripes. Half the heads perform horizontal stripe attention, half perform vertical stripe attention, and both are computed in parallel; the module is self-attention over one feature map, not cross-attention between two sources (Dong et al., 2021). AEWin extends this idea by combining local window attention with horizontal and vertical axial attention, allocating $j$5 heads to horizontal attention, $j$6 to vertical attention, and $j$7 to window attention (Zhang et al., 2022). Both mechanisms enlarge receptive field, but neither uses queries from one source and keys-values from another.

Swin-Free and Gated-SwinRMT remain closer to the shifted-window lineage. Swin-Free keeps standard non-overlapping local self-attention inside an $j$8 window but varies $j$9 across stages, using $n$0 for the $n$1 setting so that later-stage windows span multiple earlier windows without any cyclic shift or reverse-shift operations (Koo et al., 2023). Gated-SwinRMT preserves Swin’s efficient shifted-window structure, but the internal token mixer is decomposed Manhattan self-attention rather than full 2-D attention; two variants combine spatial decay with either sigmoid-based SWAT or softmax-normalized retention, and the Retention variant adds an explicit G1 gate after local context enhancement but before the output projection $n$2 (Maity et al., 7 Apr 2026). These are windowed self-attention or retention modules with locality priors, not cross-attention modules in the standard sense.

Other works replace shifted windows with non-attention or globally pooled mechanisms. FwNet-ECA keeps Window Multi-Head Self-Attention for local processing, then inserts a Fourier-domain Filter Enhancement layer that establishes a global receptive field and implicitly connects information across all spatial regions without physically shifted windows (Mian et al., 25 Feb 2025). MSCSA aggregates feature maps from multiple backbone stages by average pooling them to a common spatial size, optionally squeezing channels with a $n$3 convolution, concatenating them, and then performing attention from the original resolution to concatenated multi-scale keys and values. The paper states explicitly that this attention is “not window-restricted” and that the module is “not a standard ‘windowed cross-attention’ module” (Shang et al., 2023).

Iwin and the recent sparse sequence models reinforce the same boundary. Iwin Transformer rearranges the feature map through a Reshape–Transpose–Reshape operation so that distant tokens fall into the same attention window, then combines Interleaved Window Multi-Head Self-Attention with depthwise separable convolution in parallel. The paper treats this as global information exchange within a single module, but again it is self-attention over one source (Huo et al., 24 Jul 2025). In audio, Windowed Sink Attention replaces full temporal self-attention with a local window plus globally visible sink tokens; it remains self-attention with a structured sparsity pattern (Benetatos et al., 29 Oct 2025). GatedFWA similarly proposes a causal sliding-window self-attention mechanism with additive decay bias and FlashAttention-compatible implementation, and states explicitly that it is not about cross-attention per se (Liu et al., 8 Dec 2025).

The broader significance of these adjacent mechanisms is methodological rather than terminological. Across registration, speech enhancement, retrieval, vision backbones, and sequence modeling, windowing is repeatedly used to encode bounded interaction priors while avoiding all-pairs cost. What differentiates a true windowed cross-attention module is not merely the presence of windows, but the coexistence of three properties: distinct query and key-value sources, locality-constrained interaction, and task-specific alignment across those sources. Where one of these properties is absent, the mechanism is better described as windowed self-attention, cross-window communication, or structured sparse mixing rather than as windowed cross-attention.