---
title: Vision Mamba Models
url: https://www.emergentmind.com/topics/vision-mamba-models
type: topic
---

# Vision Mamba Models

Vision Mamba Models refer to a class of vision backbones based on state-space models (SSMs), specifically Mamba, that aim to provide efficient, scalable, and context-aware alternatives to transformer- and convolution-based visual representation learning. These architectures introduce bidirectional and selective scan mechanisms, spatiotemporal pipelines, and a set of architectural innovations that together offer linear computational and memory complexity while preserving or surpassing the accuracy of conventional models for image and video understanding tasks [2502.07161, 2401.09417, 2410.10382].

## 1. Mathematical Foundations: State-Space Models and Selectivity

At the heart of Vision Mamba is the continuous-time linear SSM, parameterized by dynamic matrices that are input-dependent and vary along the sequence and batch dimension. Formally, for a sequence of embedded tokens $x_{1:M}\in\mathbb{R}^{M\times E}$, the SSM evolves a hidden state $s_t\in\mathbb{R}^N$:

\[
s_t = A s_{t-1} + B x_t, \qquad y_t = C s_t,
\]

with $A\in\mathbb{R}^{N\times N}$, $B\in\mathbb{R}^{N\times E}$, and $C\in\mathbb{R}^{E\times N}$. In Vision Mamba, these matrices are not fixed but are generated dynamically for each token and batch element through learned projections on the input.

Specifically, for each Mamba block at layer $l$, the input tensor $T_{l-1}\in\mathbb{R}^{B\times M\times D}$ is normalized and projected to produce $x$ and gating vector $z$:

\[
T_{l-1}' = \mathrm{Norm}(T_{l-1}), \quad x = W^x T_{l-1}', \quad z = W^z T_{l-1}',
\]

where both $x,z\in\mathbb{R}^{B\times M\times E}$. Direction-specific convolutions and non-linearities extract per-time-step SSM parameters, enabling the block to modulate its recurrent dynamics selectively:

\[
x_o' = \mathrm{SiLU}(\mathrm{Conv1d}_o(x)); \quad B_o = W^{B_o} x_o', \quad C_o = W^{C_o} x_o', \quad
\Delta_o = \log(1 + \exp(W^{\Delta}_o x_o' + p_o^\Delta)).
\]

Time-varying matrices are constructed as:

\[
\bar{A}_o = \Delta_o \otimes P_o^A, \qquad \bar{B}_o = \Delta_o \otimes B_o,
\]

where $P_o^A\in\mathbb{R}^{E\times N}$ serves as a base direction-specific transition.

The selective scan is achieved through:

\[
y_o = \mathrm{SSM}(\bar{A}_o, \bar{B}_o, C_o)(x_o'),
\]

followed by gating and fusion:

\[
\tilde{y}_{\mathrm{fwd}} = y_{\mathrm{fwd}} \odot \mathrm{SiLU}(z), \quad
\tilde{y}_{\mathrm{bwd}} = y_{\mathrm{bwd}} \odot \mathrm{SiLU}(z),
\]
\[
T_l = W^T(\tilde{y}_{\mathrm{fwd}} + \tilde{y}_{\mathrm{bwd}}) + T_{l-1}.
\]

The selectivity emerges from the gating $\mathrm{SiLU}(z)$ and per-position scaling $\Delta_o$, allowing adaptive focus and computational allocation across the sequence [2502.07161].

## 2. Bidirectional, Multidimensional, and Selective Scanning

To adapt sequential SSMs to vision tasks, Vision Mamba employs various scanning strategies:

- **1D bidirectional scanning**: Each block performs forward and backward scans over the token sequence, ensuring tokens have both past and future spatial or temporal context [2401.09417].
- **2D and multi-directional scanning**: In advanced variants (e.g., VMamba, V2M), multiple flattening and scan routes are used—such as four-way (corner-to-corner) scans—to recover 2D spatial inductive bias and aggregate information along multiple axes [2410.10382, 2410.03105].
- **Parallel scan algorithms**: Efficient implementations employ parallel prefix-sum algorithms (e.g., Blelloch scan), reducing the sequential depth of the recurrence from $O(L)$ to $O(\log L)$ for sequence length $L$, as in FastVim, which further reduces depth by alternate spatial pooling [2502.00594].
- **Selective gating and dynamic adaptation**: Vision Mamba blocks use learned gates and per-position modulation, offering position-aware and input-adaptive recurrence that "filters" uninformative positions and enhances contextual integration [2502.07161].

## 3. Spatiotemporal Extensions and VideoMamba

Vision Mamba has been generalized for spatiotemporal data (e.g., video) through **VideoMamba** and related variants:

- **Temporal scan**: Recurrence across frame sequences at a fixed spatial position captures temporal dependencies.
- **Spatial scan**: Per-frame scanning along spatial dimensions enables spatial context aggregation.
- **Cross-scan and fusion modules**: Mechanisms like Structure-Aware State Fusion (SASF) and Spatial & Channel Attention (SCAttn) transfer information between parallel scan streams, enabling integration of global and local features [2502.07161].
- **Hierarchical and multiscale design**: Direction alternation and multi-scale strategies, as in Hi-Mamba and Multi-Scale VMamba, allow local patterns to be processed early and global context to be integrated later, balancing model capacity and efficiency [2405.14174].

These architectural principles collectively ensure that Vision Mamba and its descendants attain both linear complexity and robust modeling of long-range dependencies in spatiotemporal data [2410.03105, 2502.07161].

## 4. Architectural Innovations and Position Embedding

Vision Mamba introduces several architectural enhancements beyond the core SSM block:

- **Position embeddings**: Small bias terms in convolutional kernels or step parameters (e.g., $\bar{A}_o$, $\bar{B}_o$) inject positional information, critical due to the loss of explicit spatial relationships during sequence flattening [2502.07161, 2401.09417].
- **Cross-scan and fusion modules**: Modules for mixing information from different scan directions (e.g., SASF, SCAttn) are critical for fusing spatial and temporal streams and recovering 2D/3D locality [2502.07161].
- **Hierarchical and multiscale grouping**: Direction Alternation Hi-Mamba Groups (DA-HMGs) and multi-scale scanning preserve sensitivity to both local and global patterns by alternating processing at different scales and directions [2502.07161, 2405.14174].
- **Parallel, hardware-aware implementation**: All major operations in the Mamba block (matrix-vector products, convolutions) are cast as 1D convolutions and recurrences, enabling efficient CUDA kernel fusion and maximal GPU utilization [2401.09417].

These innovations are motivated by the need to balance inductive bias for spatial/temporal structure, maintain global receptive fields, and ensure hardware efficiency.

## 5. Computational and Memory Complexity

Vision Mamba achieves **linear complexity per layer** with respect to sequence length ($O(M)$ for $M$ tokens), in sharp contrast to the $O(M^2)$ complexity of transformer self-attention. This is analytically established for:

- **Bidirectional Vision Mamba blocks** (per-layer cost): Two $O(M)$ SSM recursions plus a sequence of low-cost linear or convolutional operations [2502.07161].
- **Memory footprint**: Mamba blocks only require current/past state vectors instead of full $M\times M$ attention matrices, leading to substantial savings in both FLOPs and memory [2401.09417].
- **Empirical efficiency**: ViM achieves 2.8× higher throughput and up to 86.8% less GPU memory usage compared to DeiT for high-resolution images (e.g., $1248 \times 1248$) during batch inference [2401.09417].
- **Parallel scan optimizations**: FastVim reduces scan depth by half using interleaved pooling and achieves up to 72.5% inference speedup at $2048 \times 2048$ resolution with almost no loss in accuracy [2502.00594].

## 6. Empirical Performance and Ablative Findings

Across core computer vision benchmarks, Vision Mamba and its variants consistently match or surpass the accuracy of attention-based backbones at lower compute:

| Model     | Params (M) | ImageNet Top-1 (%) | Throughput (img/s) | COCO box AP | ADE20K mIoU |
|-----------|------------|--------------------|--------------------|-------------|-------------|
| Vim-Tiny  | 7          | 72.8               | 1,050              | –           | –           |
| Vim-Small | 26         | 79.3               | 2,100              | 46.7        | –           |
| Vim-Base  | 98         | 82.1               | 1,500              | –           | 50.5        |
| ViT-Base  | 86         | 81.8               | 600                | –           | 48.7        |

- **Bidirectionality and gating**: Bidirectional scanning offers a 0.5–0.8% gain over unidirectional, and learned gating provides additional 0.3–0.6% top-1 accuracy, with significant improvements in dynamic focus for moving objects in video [2502.07161].
- **Ablation of key architectural components**: Removing cross-scan fusion or hierarchical alternation degrades action recognition and reduces robustness to view changes; diminishing returns are observed beyond a certain number of hierarchical layers [2502.07161].
- **Positional embeddings**: Learnable 2D embeddings outperform fixed or 1D embeddings, with a typical 0.8% top-1 improvement on ImageNet [2401.09417].
- **Scaling**: Vision Mamba models scale to high resolutions and longer token sequences without the throughput collapse seen in traditional ViTs, retaining graceful degradation in accuracy.

## 7. Extensions, Limitations, and Future Directions

Advancements and ongoing directions related to Vision Mamba include:

- **Spatiotemporal and multimodal pipelines**: Extensions to video (VideoMamba), video-language, and low-level tasks (segmentation, dense prediction) through tailored scan/fusion modules and multiscale design [2502.07161, 2410.03105].
- **Model compression and acceleration**: Dynamic Vision Mamba (DyVM) introduces token and block redundancy reduction via training-aligned pruning and dynamic block selection, achieving up to 35% FLOPs reduction with minimal accuracy loss [2504.04787]. FastVim's pooling strategies further accelerate inference [2502.00594].
- **Stability and inductive bias**: Adaptive scan strategies and positionally-aware embeddings aim to resolve the loss of spatial priors and causality mismatches in classic SSM-to-vision adaptations [2404.18861].
- **Interpretability, generalization, robustness**: Open challenges persist in understanding the interpretability of SSM state evolution, domain generalization to out-of-distribution data, adversarial robustness involving dynamically learned B,C,Δ parameters, and theoretical scaling laws [2404.18861, 2410.03105].
- **Hybrid architectures**: Integration of convolutional, attention-based, and SSM modules (e.g., MambaVision) seeks to harness complementary inductive biases for edge and global structure extraction [2407.08083].
- **Next-generation models**: Research is aiming towards SSM–attention fusion, high-dimensional scan patterns, unified multi-modal architectures (including video and language), and further hardware and distributed system optimization [2502.07161, 2410.03105].

Vision Mamba thus represents a rapidly maturing paradigm for scalable, context-aware visual representation learning, with demonstrable benefits in linear complexity, global context integration, and experimental performance across the full spectrum of visual tasks [2502.07161, 2401.09417, 2410.10382, 2410.03105].

Source: https://www.emergentmind.com/topics/vision-mamba-models