Physics-Inspired Sliding Attention

• Physics-inspired sliding attention is a design paradigm that replaces all-to-all interactions with finite-range, geometry-aware local windows that move with the query.
• It employs sliding window and boundary adjustment strategies to preserve training-scale receptive fields, thus balancing fine-grained detail with overall coherence.
• These methods are successfully applied in video generation, compression, and molecular docking, offering efficient and scalable attention mechanisms.

Physics-inspired sliding attention denotes a family of attention mechanisms that replace unconstrained all-to-all interactions with finite-range, geometry-aware, or scale-aware interaction rules that move with the query in sequence space, on regular lattices, or in continuous metric space. In the recent literature, the sliding component appears as inward sliding windows that preserve the training-scale receptive field in video Diffusion Transformers, Gaussian distance kernels over learnable 1D residue positions, patchless 3D local attention on latent video volumes, tile-level block-sparse windows aligned with FlashAttention blocks, approximate $k$-NN neighborhoods in continuous ECEF coordinates, multiscale window attention coupled through coarse grids, and causal sparse bands derived from ordered Markov boundaries in lattice spin systems (Wu et al., 18 Nov 2025, Fan, 5 Jan 2026, You et al., 27 Sep 2025, Colagrande et al., 3 Jul 2025, Zhong et al., 22 Jun 2026).

1. Conceptual scope and recurrent design motifs

A common starting point is the cost profile of standard self-attention. In video Diffusion Transformers, a latent video with spatial size $H \times W$ and temporal length $F$ contains $N = H \times W \times F$ tokens, and standard scaled dot-product attention

$$\mathrm{Attn}(Q,K,V)=\mathrm{Softmax}\left(\frac{QK^\top}{\sqrt{d}}\right)V$$

has time and memory complexity $\mathcal{O}(N^2 d)=\mathcal{O}((HWF)^2 d)$ (Wu et al., 18 Nov 2025). In image transformers, global self-attention on an $H \times W$ input scales as $O((HW)^2)$, which motivated local attention variants that restrict each query to neighboring pixels (Pan et al., 2023). In latent video compression, the analogous issue appears on a latent tensor $y \in \mathbb{R}^{L \times H \times W \times C}$, where local 3D attention is adopted to keep the entropy model practical (Kopte et al., 4 Oct 2025).

Across these settings, several motifs recur. One is finite interaction distance: attention is restricted to a local neighborhood whose size is fixed by a training scale, a bandwidth parameter, a tile window, or an ordered Markov boundary. A second is boundary handling: some methods truncate windows at boundaries, while others shift the window inward to preserve interaction volume. A third is global correction or coarse guidance: local interactions alone often improve detail or efficiency but can lose long-range coherence, so several works add a global branch, a multiscale hierarchy, or a coarse representation. A fourth is hardware alignment: sliding attention is often reformulated so that sparse logical structure still maps to dense GPU blocks or standard convolutional primitives (Wu et al., 18 Nov 2025, Pan et al., 2023, Zhang et al., 6 Feb 2025).

This suggests that the phrase does not identify a single operator. It identifies a design regime in which locality, geometry, scale separation, and computational structure are made explicit.

2. Finite interaction range, sliding windows, and boundary conditions

In the ultra-high-resolution video setting of FreeSwim, the core hypothesis is that each token at inference must see approximately the same spatial neighborhood size it saw during training, namely the training-scale receptive field (Wu et al., 18 Nov 2025). Let the pretrained model’s native spatial size be $h \times w$, and the high-resolution target be $H \times W$0. FreeSwim enforces that each query token at high resolution still only interacts with a window of size $H \times W$1, even though the canvas is larger. This is the sliding-window mechanism.

The distinctive feature is inward sliding. A centered window is used for interior points, but when that window would extend beyond the boundary, the window is shifted inward so that its size remains exactly $H \times W$2 while staying fully within the larger domain. The binary mask $H \times W$3 is defined through offsets

$H \times W$4

$H \times W$5

followed by a rectangular admissibility condition in $H \times W$6 and masked attention

$H \times W$7

The stated purpose is to preserve the same number of interactable tokens for all queries, including boundary points (Wu et al., 18 Nov 2025).

A different sparse sliding construction appears in FlashVAN for spin glasses. There, the width of the causal sliding window is derived from the ordered Markov boundary $H \times W$8, the minimal subset of previous variables such that $H \times W$9. On $F$0-dimensional lattices with raster-scan ordering, the maximum index distance scales as $F$1 in 1D, $F$2 in 2D, and $F$3 in 3D, leading to a causal band mask of the form

$F$4

The resulting complexity becomes $F$5 instead of $F$6, with $F$7 chosen from the physical Markov structure rather than as an arbitrary truncation (Zhong et al., 22 Jun 2026).

By contrast, the 3D SWA entropy model for learned video compression uses a patchless local cuboid on the latent lattice and explicitly truncates the window at boundaries rather than shifting it inward. A head-specific bias entry is

$F$8

with additional causal masking over the line-scan order (Kopte et al., 4 Oct 2025). The paper emphasizes that no zero padding is used; interactions simply stop at the boundary.

These variants expose a technical point that is often obscured in generic discussions of local attention. “Sliding” is not just window restriction. It includes a choice of how the interaction domain itself moves and deforms near boundaries, and that choice changes the effective prior.

3. Geometry-grounded kernels and latent docking dynamics

A second line of work interprets sliding attention through explicit distance kernels. In Adaptive Density Fields, attention is formulated directly in continuous metric space. Points of interest are $F$9 in ECEF coordinates, a query is $N = H \times W \times F$0, and similarity is induced by physical distance rather than learned latent projections. For approximate $N = H \times W \times F$1-NN neighbors $N = H \times W \times F$2, the kernel is

$N = H \times W \times F$3

with adaptive bandwidth

$N = H \times W \times F$4

The resulting field is

$N = H \times W \times F$5

and FAISS $N = H \times W \times F$6-NN is treated as top-$N = H \times W \times F$7 attention pruning intrinsic to the operator (Fan, 5 Jan 2026). The paper explicitly identifies the query as location, the keys as POI positions, the values as scalar scores, and the kernel as the unnormalized attention weight.

ABConformer adopts a different geometry: a learnable 1D positional space that mimics a docking process for antibody–antigen interface prediction from sequence alone. Let $N = H \times W \times F$8 denote the sliding sequence and $N = H \times W \times F$9 the reference sequence. Feature affinity is

$$\mathrm{Attn}(Q,K,V)=\mathrm{Softmax}\left(\frac{QK^\top}{\sqrt{d}}\right)V$$0

where $$\mathrm{Attn}(Q,K,V)=\mathrm{Softmax}\left(\frac{QK^\top}{\sqrt{d}}\right)V$$1 is the usual scaled dot product after learnable projections, while spatial affinity is

$$\mathrm{Attn}(Q,K,V)=\mathrm{Softmax}\left(\frac{QK^\top}{\sqrt{d}}\right)V$$2

The joint unnormalized weight is

$$\mathrm{Attn}(Q,K,V)=\mathrm{Softmax}\left(\frac{QK^\top}{\sqrt{d}}\right)V$$3

followed by row- and column-normalized versions $$\mathrm{Attn}(Q,K,V)=\mathrm{Softmax}\left(\frac{QK^\top}{\sqrt{d}}\right)V$$4 and $$\mathrm{Attn}(Q,K,V)=\mathrm{Softmax}\left(\frac{QK^\top}{\sqrt{d}}\right)V$$5. The crucial “sliding” step is the position update

$$\mathrm{Attn}(Q,K,V)=\mathrm{Softmax}\left(\frac{QK^\top}{\sqrt{d}}\right)V$$6

so each sliding residue is pulled toward a weighted barycenter of reference positions (You et al., 27 Sep 2025). The paper characterizes this as a mean-shift / gradient ascent process on an interaction density, and the physical analogy is molecular docking in which contacts are distance-limited and localized.

The contrast between these two formulations is instructive. ADF is explicitly geometry-preserving and metric-grounded in physical coordinates, with no learned similarity metric in the kernel. ABConformer instead uses a learned latent positional coordinate and a Gaussian prior over that coordinate. Both are physics-inspired, but they instantiate different meanings of “physics”: one is continuous-space field evaluation; the other is docking-like local interaction dynamics without explicit 3D structure.

4. Multiscale and global-context mechanisms

Strict locality is often insufficient. FreeSwim reports that inward sliding-window attention alone greatly improves fine-grained details but produces repetitive objects and tiling-like artifacts; full attention alone preserves global semantics but yields poor local detail quality at unseen high resolution (Wu et al., 18 Nov 2025). Its response is a dual-path pipeline. A window branch uses inward sliding-window self-attention, while a full branch uses full 3D self-attention. The branches are coupled through cross-attention to text, and the window branch’s cross-attention output is replaced by the full branch’s output: $$\mathrm{Attn}(Q,K,V)=\mathrm{Softmax}\left(\frac{QK^\top}{\sqrt{d}}\right)V$$7 The reported best choice is $$\mathrm{Attn}(Q,K,V)=\mathrm{Softmax}\left(\frac{QK^\top}{\sqrt{d}}\right)V$$8, i.e. pure override. To reduce cost, the full branch is refreshed only every $$\mathrm{Attn}(Q,K,V)=\mathrm{Softmax}\left(\frac{QK^\top}{\sqrt{d}}\right)V$$9 denoising steps, leading to effective complexity

$\mathcal{O}(N^2 d)=\mathcal{O}((HWF)^2 d)$0

Empirically, $\mathcal{O}(N^2 d)=\mathcal{O}((HWF)^2 d)$1 gives $\mathcal{O}(N^2 d)=\mathcal{O}((HWF)^2 d)$2 speedup at 1080P with negligible or even slightly improved metrics, and at 4K the paper reports up to $\mathcal{O}(N^2 d)=\mathcal{O}((HWF)^2 d)$3 speedup for $\mathcal{O}(N^2 d)=\mathcal{O}((HWF)^2 d)$4 (Wu et al., 18 Nov 2025).

MANO addresses the same local/global tension through a multiscale hierarchy rather than a parallel global branch. The finest-scale feature map $\mathcal{O}(N^2 d)=\mathcal{O}((HWF)^2 d)$5 is repeatedly downsampled by a shared convolution,

$\mathcal{O}(N^2 d)=\mathcal{O}((HWF)^2 d)$6

local or window attention is applied at each scale, and the outputs are upsampled and summed: $\mathcal{O}(N^2 d)=\mathcal{O}((HWF)^2 d)$7 Because coarse windows correspond to large physical neighborhoods on the original grid, the method behaves like a multipole or near-/far-field decomposition. For fixed window size $\mathcal{O}(N^2 d)=\mathcal{O}((HWF)^2 d)$8, the total complexity is

$\mathcal{O}(N^2 d)=\mathcal{O}((HWF)^2 d)$9

linear in the number of grid points $H \times W$0 (Colagrande et al., 3 Jul 2025).

This suggests two distinct but closely related strategies for restoring global coherence in sliding attention. One strategy keeps a local path and adds an explicit global field. The other uses repeated local computations on coarser grids so that distant interactions are mediated through coarse summaries. Both preserve local inductive bias while avoiding a purely myopic receptive field.

5. Representative domains and reported behavior

The literature spans video generation, video compression, antibody–antigen interface prediction, scalable spatial modeling, vision backbones, PDE operators, and spin-glass sampling. The following examples illustrate how the same design language is specialized in different technical regimes.

Setting	Mechanism	Reported outcome
Ultra-high-resolution video generation	FreeSwim: inward sliding-window self-attention, dual-path full/window branches, cross-attention override, caching (Wu et al., 18 Nov 2025)	superior performance on VBench; $H \times W$1 gives $H \times W$2 speedup at 1080P
Sequence-only Ab–Ag interface prediction	ABConformer: Gaussian distance kernel over learnable 1D positions with iterative position updates (You et al., 27 Sep 2025)	epitope IoU $H \times W$3 vs AF2 Multimer $H \times W$4; paratope IoU $H \times W$5 vs $H \times W$6
Hierarchical vision backbones	Slide Attention: local self-attention implemented with depthwise convolution and deformed shifting (Pan et al., 2023)	Slide-Swin-T $H \times W$7 top-1, $H \times W$8 FPS; Swin-T $H \times W$9
Physics and vision on grids	MANO: multiscale window attention with coarse-grid aggregation (Colagrande et al., 3 Jul 2025)	linear time and memory complexity with global receptive field
Spin-glass Boltzmann sampling	FlashVAN: sparse causal sliding attention from ordered Markov boundaries, FlashAttention, KV cache (Zhong et al., 22 Jun 2026)	up to two orders of magnitude speedup over vanilla variational autoregressive networks
Learned video compression	Patchless 3D SWA with masked $O((HW)^2)$0 decoder-only attention (Kopte et al., 4 Oct 2025)	up to $O((HW)^2)$1 BD-rate savings; decoder complexity reduced by a factor of $O((HW)^2)$2
Fast video diffusion inference	STA: tile-level 3D sliding windows aligned with FlashAttention blocks (Zhang et al., 6 Feb 2025)	attention $O((HW)^2)$3-$O((HW)^2)$4 over FA2 and $O((HW)^2)$5-$O((HW)^2)$6 over FA3; $O((HW)^2)$7s to $O((HW)^2)$8s training-free

Several empirical patterns recur. In ABConformer, replacing sliding attention with conventional cross-attention increases recall but lowers precision and PR-AUC, which the paper interprets as more false positives from diffuse global coupling (You et al., 27 Sep 2025). In FreeSwim, local-only window attention improves detail yet repeats objects, whereas full attention is coherent but blurry at high resolution (Wu et al., 18 Nov 2025). In learned video compression, longer temporal context initially improves rate–distortion performance but excessive context degrades it; the best context lengths reported are $O((HW)^2)$9 frames on HEVC B and $y \in \mathbb{R}^{L \times H \times W \times C}$0 frames on UVG, while full $y \in \mathbb{R}^{L \times H \times W \times C}$1-frame context incurs a BD-rate penalty of $y \in \mathbb{R}^{L \times H \times W \times C}$2 and $y \in \mathbb{R}^{L \times H \times W \times C}$3, respectively (Kopte et al., 4 Oct 2025). In STA, a window covering only $y \in \mathbb{R}^{L \times H \times W \times C}$4 of tokens accounts for $y \in \mathbb{R}^{L \times H \times W \times C}$5 of the attention scores on average in pretrained HunyuanVideo, and head-wise locality patterns are sufficiently stable to support per-head mask search (Zhang et al., 6 Feb 2025).

These results support a limited generalization. Sliding attention tends to be most effective when the target dependency structure is strongly local or multiscale, and when the mechanism used to recover long-range effects is matched to the domain.

6. Misconceptions, limitations, and open directions

A common misconception is that “physics-inspired” implies explicit simulation of physical laws. The papers use the term more broadly. In FreeSwim, the physics language concerns finite interaction range, reflecting or shifted boundary conditions, and a coarse global field (Wu et al., 18 Nov 2025). In ABConformer, it denotes a 1D docking-like abstraction rather than explicit 3D molecular geometry (You et al., 27 Sep 2025). In ADF, it denotes metric-grounded kernels in ECEF space and zones of influence derived from trajectory-conditioned fields (Fan, 5 Jan 2026). In FlashVAN, it is tied to conditional independence, lattice geometry, and the ordered Markov boundary of an autoregressive factorization (Zhong et al., 22 Jun 2026).

Another misconception is that locality by itself is always beneficial. Multiple papers report the opposite. Pure local windows in FreeSwim produce repetition. Pure local attention in MANO degrades on higher-resolution Darcy problems because long-range effects are not captured. In learned video compression, excessive temporal context can also degrade performance, indicating that the issue is not simply “more context is better” but rather how context is structured and filtered (Colagrande et al., 3 Jul 2025, Kopte et al., 4 Oct 2025).

The limitations are correspondingly heterogeneous. FreeSwim notes the cost of the global branch even with caching, leaves temporal locality unexplored, and fixes the window size to the training scale (Wu et al., 18 Nov 2025). ADF assumes static geometry, uses isotropic kernels, and is presented more as a post-hoc analytic field than as a differentiable end-to-end layer (Fan, 5 Jan 2026). ABConformer uses scalar positions $y \in \mathbb{R}^{L \times H \times W \times C}$6, so long-range or multi-patch epitopes may be under-represented in the 1D sliding frame (You et al., 27 Sep 2025). Slide-Transformer retains the general limitation that pure local attention does not directly capture long-range dependencies, even if hierarchical design mitigates this (Pan et al., 2023). The learned video compression model remains fully sequential in line-scan decoding, so theoretical locality gains do not automatically yield maximal wall-clock decoding speed (Kopte et al., 4 Oct 2025). STA addresses hardware inefficiency in 2D and 3D sliding attention, but it still assumes that most useful interactions are local within a single layer; if a task requires strong within-layer non-locality, window size or training may become critical (Zhang et al., 6 Feb 2025).

The open directions stated or implied in these works are relatively consistent. They include anisotropic kernels and full covariance matrices in continuous-space attention, temporal sliding windows or causal propagation in video generators, hierarchical or low-resolution replacements for expensive global branches, adaptive or learned bandwidth mappings, multi-scale receptive fields, and sparse patterns derived from explicit physical or graphical conditional-independence structure rather than from generic top-$y \in \mathbb{R}^{L \times H \times W \times C}$7 heuristics (Fan, 5 Jan 2026, Wu et al., 18 Nov 2025, Zhong et al., 22 Jun 2026).

Taken together, the literature indicates that physics-inspired sliding attention is best understood as a principled biasing of attention toward local, metric, or multiscale interaction laws. The principal technical question is not whether to abandon global attention categorically, but how to encode locality, boundary conditions, and coarse global structure so that the induced interaction pattern matches both the domain geometry and the computational substrate.