Visual State Space Duality (VSSD)

Updated 12 November 2025

Visual State Space Duality (VSSD) is a neural paradigm that reinterprets state-space models for non-causal, global token mixing in vision tasks.
It utilizes forward and backward scans to eliminate causality, replacing quadratic self-attention with efficient O(N) operations for tasks like image classification and segmentation.
Recent variants like EfficientViM compress channel mixing into smaller hidden states, yielding significant computational savings and improved empirical performance.

Visual State Space Duality (VSSD) is a paradigm for constructing neural sequence models, particularly in computer vision, that leverages a non-causal reinterpretation of State Space Models (SSMs), allowing for fully parallel, linear-time global token mixing. VSSD emerges from the state-space duality (SSD) framework developed in Mamba2, extending it from strictly causal to non-causal processing, which is critical for image understanding tasks where bidirectional context is needed. Recent variants, including EfficientViM, further optimize SSD by compressing costly sequence-wide channel mixing into small hidden-state spaces, resulting in significant computational and memory reductions while preserving or enhancing empirical performance.

1. Mathematical Foundations

VSSD is formulated by discarding the magnitude of token-to-hidden interactions in the standard SSD recurrence, preserving only relative weights, and aggregating results from both forward and backward “scans” to eliminate causality. Denote an input sequence $x(1 ... L)$ and per-token parameters $A_t \in \mathbb{R}$ , $B_t \in \mathbb{R}^D$ , $C_t \in \mathbb{R}^D$ . The standard SSD update is: $h(t) = A_t h(t-1) + B_t x(t), \qquad y(t) = C_t^\top h(t)$ Unrolled, this defines a causal quadratic-form convolution: $y_j = \sum_{i=1}^j C_j^\top (A_jA_{j-1}\cdots A_{i+1}) B_i x_i$ To remove causality, VSSD reinterprets $A_t$ as a scaling for $x(t)$ ’s contribution and discards its magnitude in recurrence: $h(t) = h(t-1) + \frac{1}{A_t} B_t x(t) \implies h(t) = \sum_{i=1}^t \frac{1}{A_i} B_i x(i)$ By aggregating both forward and backward scans, the global hidden state becomes: $H = \sum_{j=1}^L m_j Z_j, \qquad Z_j = B_j x(j), \quad m_j = \frac{1}{A_j}$ All tokens observe identical context, yielding a non-causal, permutation-invariant global state.

From an implementation standpoint, the computation is reduced to a series of learned per-token contractions: $Z = X \odot B;\qquad H = m^\top Z;\qquad Y = C H$ This can also be reformulated as a kind of linear attention: $Y = C (B^\top (X \odot m))$

2. Architectural Composition

VSSD is instantiated as a four-stage hierarchical vision backbone, resembling Swin Transformer and ConvNeXt at a macro level, but with almost all self-attention blocks in the early stages replaced by NC-SSD (non-causal SSD) blocks.

Stem: Overlapping $3 \times 3$ convolutions with stride 2.
Stages 1–3: Each starts with an overlapping downsampling convolution, followed by $N$ $N$ repetitions of the "VSSD block," which itself consists of:
1. Local Perception Unit (LPU): Depthwise convolution (kernel=3) plus nonlinearity.
2. NC-SSD token mixer: As above, applied in parallel to the entire sequence.
3. Channel Feed-Forward Network (FFN): Two linear layers with GELU activation.
4. Residual connections and Layer Normalization around both LPU/NC-SSD and FFN.
Stage 4: Same downsampling as prior stages, then $K$ layers of standard multi-head self-attention (MSA) and FFN.

A schematic pseudocode for a VSSD block (omitting batch and spatial details):

for each block in stage:
    X = LayerNorm(X)
    U = DWConv3x3(X)  # Local Perception Unit
    Z = U * B         # B: learned per-token projection
    H = sum_t m_t * Z_t  # m: learned per-position weights
    Y = C(H)          # C: learned per-head readout
    X = X + Y         # Residual
    X = X + FFN(LayerNorm(X))  # FFN with residual

The overall structure allows VSSD to replace $O(N^2)$ attention blocks with $O(N)$ NC-SSD blocks, except at the final stage where self-attention can be optionally retained for marginal accuracy gains.

3. Computational and Memory Complexity

The VSSD token mixer achieves significant reductions in computational complexity:

Model Type	Complexity per Block	Parallelizable over $N$ ?
Causal SSD	$O(N D^2)$	No (sequential)
ViT (Self-attn)	$O(N^2 D)$	Yes
VSSD (NC-SSD)	$O(N D)$	Yes

In VSSD, NC-SSD involves contracting tensors over $N$ tokens and $D$ channels with learned vectors, requiring only $O(N D)$ computation per block, fully parallel over sequence length. In contrast, causal SSD requires sequential updates, and self-attention scales quadratically with sequence length. The method is fully parallelizable, offering hardware efficiency, especially for long sequences or high-resolution vision data.

Recent approaches such as EfficientViM further compress the most expensive channel mixing into a small hidden-state dimension $N \ll D$ , dropping the $O(L D^2)$ cost down to $O(N D^2)$ per block, while the main global token contraction remains linear.

4. Empirical Benchmarking

Extensive experiments on ImageNet-1K, COCO, and ADE20K establish VSSD’s performance edge relative to SSM and transformer variants. Key condensed results (abridged from tables):

Image Classification (ImageNet-1K, Top-1 Accuracy):

Model	Type	Params (M)	Top-1 (\%)
ConvNeXt-T	Conv	29	82.1
Swin-T	Attn	29	81.3
VMambaV9-T	SSM	31	82.5
VSSD-T	SSD	24	83.7

VSSD-T (24M parameters) achieves a top-1 accuracy of 83.7%, representing a 1.2% gain over the best SSM baseline and reduced parameter count.

Object Detection and Segmentation (COCO, Mask R-CNN):

Backbone	AP $^b$	AP $^m$	Params (M)
Swin-T	42.7	39.3	48
VMamba-T	46.5	42.1	42
VSSD-T	46.9	42.6	44

Semantic Segmentation (ADE20K, UPerNet):

Backbone	mIoU (\%)	Params (M)
Swin-T	44.4	60
VMamba-T	47.3	55
VSSD-T	47.9	53

VSSD-T is consistently superior or comparable to SSM-based and transformer baselines with lower parameter or FLOP counts. Additionally, VSSD achieves 20–50% faster training throughput over vanilla SSD, and hybridizing with self-attention in the last stage yields a marginal accuracy boost (+0.2%) at negligible computational cost.

5. Role of the Non-Causal Mixer and Ablations

Replacing the causal SSD operator with NC-SSD is critical for both non-causal vision tasks and computational efficiency. Empirical ablations show that the learned per-token vector $m$ is indispensable: removing it leads to model collapse during training. The non-causal formulation (via multi-scan strategies and global hidden-state sharing) ensures all spatial tokens obtain equivalent, global receptive fields, unlike causal SSMs, which are limited by strict unidirectional context.

In EfficientViM, the "hidden-state-mixer" (HSM-SSD) further reduces runtime by relocating channel mixing from the full $L\times D$ sequence dimension into the compact hidden state dimension, maintaining $O(L N D + N D^2)$ complexity and facilitating resource-efficient inference and training. The multi-stage hidden-state fusion introduced in EfficientViM supports enhanced representation power at each intermediate layer, with negligible overhead and improved accuracy.

6. Broader Implications and Applications

VSSD constitutes a lightweight, hardware-efficient alternative to quadratic self-attention for image classification, object detection, and semantic segmentation. Its non-causal, global-mixing mechanism achieves full token interactivity without sacrificing parallelism, enabling applicability to longer sequences and higher-resolution images than previously feasible with attention or causal SSMs. EfficientViM demonstrates that further architectural refinements, such as hidden-state-mixer-based token aggregation and multi-stage fusion, can push the speed-accuracy trade-off to Pareto-optimal levels on standard vision benchmarks, achieving up to 0.7% improvement over previous architectures with increased throughput and scalability to high-resolution settings (Shi et al., 26 Jul 2024, Lee et al., 22 Nov 2024).

A plausible implication is that state-space models, equipped with non-causal dual forms and hidden-state mixing, may generalize as a primary replacement for attention in a broad array of sequence processing tasks beyond vision. The clear separation between local feature extraction (via convolutional LPUs) and global context mixing (via VSSD) offers an explicit architectural modularity useful in both research and deployment environments.

7. Limitations and Research Directions

The empirical superiority of VSSD relies on a carefully balanced parameterization (e.g., the handling of $m$ and choice of hidden state dimensions in HSM-SSD). Instabilities can occur if key components—such as the per-token scaling vector $m$ —are ablated. While the global, parallel computation is efficient, the outright discard of magnitude information may not be optimal for all domains, particularly if domain knowledge favors directional or temporal structure. In EfficientViM, pushing all channel mixing into the hidden dimension succeeds as long as $N \ll D$ , but performance—especially on highly non-local tasks—may degrade if this assumption does not hold.

Future research may address the expressivity gap between non-causal SSDs and richer but more computationally expensive transformers, explore further applications of the VSSD/HSM-SSD principle in non-vision modalities, and investigate hybrid models partitioning global and local context more flexibly.

In conclusion, Visual State Space Duality provides a scalable, theoretically principled framework for efficient global context modeling in vision, establishing a core methodological advance in the design of sequence operators for high-throughput deep learning workloads.