Vision Mamba State-Space Encoder

Updated 13 December 2025

Vision Mamba is a visual backbone that extends 1D state-space models to 2D domains, providing global receptive fields and structured locality for image and video data.
It employs dual orthogonal scanning with multi-directional rotations to aggregate spatial information, outperforming traditional transformer-based self-attention with linear complexity.
Benchmarked on datasets like ImageNet and COCO, the encoder demonstrates hardware-efficient parallel processing and competitive accuracy, making it a scalable choice for various visual tasks.

The Vision Mamba State-Space Encoder (often abbreviated as Vision Mamba or V2M in the context of 2D models) represents a family of visual backbone architectures that generalize state-space models (SSMs) to high-dimensional visual domains. Its core innovation is the integration of dynamic, input-conditioned discrete-time SSMs for modeling long-range dependencies in image or video data with hardware-efficient linear complexity. By extending SSMs from one-dimensional (1D) sequences to two-dimensional (2D) grids, Vision Mamba architectures offer global receptive fields, inherent scalability, and structured locality priors superior to transformer-based self-attention in both throughput and spatial inductive bias.

1. Mathematical Formulation: 2D State-Space Model

Vision Mamba generalizes the standard 1D discrete-time SSM to 2D spatial domains for image encoding. For an input image with spatial tokens $x_{i,j}\in \mathbb{R}^d$ (row $i$ , column $j$ , embedding dimension $d$ ), the 2D SSM recurrence is:

$s_{i,j} = A_\text{row} \cdot s_{i-1,j} + A_\text{col} \cdot s_{i,j-1} + B \cdot x_{i,j}$

$y_{i,j} = C \cdot s_{i,j} + D \cdot x_{i,j}$

Here, $A_\text{row}, A_\text{col}\in \mathbb{R}^{N \times N}$ are the recurrence matrices propagating hidden states vertically and horizontally, respectively, $B\in \mathbb{R}^{N \times d}$ performs input-to-state projection, $C\in \mathbb{R}^{d \times N}$ is the state-to-output projection, and $D$ is an optional direct input bypass. This 2D formulation enables genuine spatial context modeling rather than artificially flattening the image to sequence form (Wang et al., 14 Oct 2024).

2. Block Architecture: Directional Scanning and Aggregation

A Vision Mamba block operates on tensors of shape $(B \times H \times W \times D)$ , where $(H, W)$ denotes the spatial grid. V2M decomposes the 2D SSM update into two orthogonal 1D scan passes (row-wise then column-wise), leveraging the highly efficient Mamba parallel scan algorithm. For optimal spatial coverage, the feature map is rotated into four directions (0°, 90°, 180°, 270°), producing four tensors. Each is independently processed through the 2D SSM (stacked along the batch axis), and the outputs are rotated back and summed element-wise. A standard feed-forward MLP with two linear layers and GeLU activation follows, with residual connections and (optionally) pre-/post-layer normalization (Wang et al., 14 Oct 2024).

Direction	Scan Paths	Aggregation
0°, 90°, 180°, 270°	Row-wise, Col-wise recursively	Sum outputs after re-rotating

This block design preserves spatial locality and efficiently propagates information from all four corners.

3. Hardware-Efficient Parallelism and Memory Layout

Mamba’s parallel scan algorithm, originally developed for 1D linear recurrences, is directly adapted for efficient 2D SSM evaluation in Vision Mamba. Feature maps are laid out such that each spatial axis (row or column) is contiguous in memory, facilitating parallel computation. At training time, batch-wise computation of directional states enables modern GPU hardware utilization rates exceeding 90%, leveraging fast on-chip SRAM buffers for both forward and backward passes. Despite computing four directional passes (which nominally doubles compute vs. single 1D scan), throughput matches 1D Mamba blocks due to parallelization and memory optimizations (Wang et al., 14 Oct 2024).

4. Training Configuration and Model Families

Vision Mamba architectures are trained as general-purpose visual backbones. Key configurations for V2M (Wang et al., 14 Oct 2024):

Pretraining: ImageNet-1K, 300 epochs, 224×224 inputs, standard augmentations.
Optimizer: AdamW, LR= $5 \times 10^{-4}$ , cosine schedule, weight decay 0.05, batch size 512.
Fine-tuning: Object detection and segmentation (COCO, ADE20K) with Mask R-CNN/UPerNet heads.
Variants: V2M-T (Tiny), V2M-S (Small), V2M-B (Base), varying in parameter count, FLOPs, and token directions (2 or 4).

Vision Mamba variants are also found in other works: VMamba (SS2D cross-scan), EfficientViM (hidden state mixer), HRVMamba (DVSS block), Multi-Scale VMamba (MSVSS block), and others (Liu et al., 18 Jan 2024, Lee et al., 22 Nov 2024, Zhang et al., 4 Oct 2024, Shi et al., 23 May 2024).

5. Empirical Performance and Benchmarking

Vision Mamba models consistently outperform or match state-of-the-art vision backbones in image classification, detection, and segmentation at lower computational cost. Representative results for V2M (Wang et al., 14 Oct 2024):

Model	Params (M)	FLOPs (G)	ImgNet Top-1 (%)	COCO Box AP	ADE20K mIoU
V2M-T	7	1.9	76.2	–	–
V2M-S*	30–62	5.4–8.2	82.9–83.8	47.6–48.9	48.2–50.8
VMamba-T	50	8.7	82.6	47.3	47.9

Ablation studies reveal gains from increased scan directions and local-window variants; the “center grid” token arrangement yields best accuracy (Wang et al., 14 Oct 2024). Across multi-directional SSM backbones, Vision Mamba maintains linear scaling in input size and robust performance for high-resolution images.

6. Inductive Biases and 2D Priors

Unlike strictly 1D flattened SSMs, Vision Mamba preserves spatial inductive biases by modeling token interactions along both axes. Multi-directional scanning further enables aggregation of local and global context, countering the loss of spatial adjacency present in naive 1D flattening. Local window SSM sub-blocks inject additional locality priors at negligible computational cost (Wang et al., 14 Oct 2024). In HRVMamba, deformable convolutional kernels address long-range forgetting and support multi-scale aggregation (Zhang et al., 4 Oct 2024). Structure-aware fusion via spatial convolution, as in Spatial-Mamba, merges sequential and spatial dependencies for effective context propagation (Xiao et al., 19 Oct 2024).

7. Implementation Considerations and Practical Guidance

Practitioners are advised to balance accuracy and compute cost by selecting model variants and scan directionality to match task requirements. Hyperparameter recommendations for V2M include:

Use two directions and Tiny variant for low-latency settings (<2G FLOPs).
Four directions and full-scale models for maximum accuracy (>8G FLOPs).
Add local windows for an extra 0.1–0.2% accuracy if memory allows.
Ensure optimal feature map layout and batched directional scans for hardware efficiency.

Vision Mamba blocks can be stacked in pyramid or flat backbones, with unaltered training and optimization recipes from existing Mamba implementations. GPU utilization is maximized through contiguous spatial layouts and batchwise stacking of orientations (Wang et al., 14 Oct 2024).

8. Research Impact and Theoretical Context

Vision Mamba represents a decisive step in bridging the gap between global context modeling and spatial locality in visual representation learning. By integrating high-dimensional, input-adaptive SSMs with hardware-aware algorithms, it achieves linear scaling and competitive accuracy on visual benchmarks. Its architecture provides a principled alternative to self-attention with mathematically interpretable recurrences. Contemporary surveys and empirical studies (Rahman et al., 4 Oct 2024, Xu et al., 29 Apr 2024, Liu et al., 7 May 2024) situate Vision Mamba at the forefront of scalable vision backbone research, with ongoing development in multi-modal, video, point cloud, and specialized domains.

In summary, the Vision Mamba State-Space Encoder (V2M and related architectures) exploits genuine 2D state-space recurrences, multi-directional scanning, and input-adaptive kernels for efficient visual representation learning, consistently surpassing prior SSM-based models in accuracy, locality preservation, and throughput (Wang et al., 14 Oct 2024).