MambaVF: Scalable SSM Architectures

• MambaVF is a family of neural architectures leveraging state-space models for efficient video fusion and multi-modal vision processing.
• The framework eliminates explicit optical flow estimation by using spatio-temporal bidirectional SSM modules, reducing complexity compared to Transformer attention.
• Comparative analyses show MambaVF achieves up to 92% parameter reduction and faster latency, making it ideal for scalable high-resolution video applications.

MambaVF refers to a family of neural architectures that leverage state space models (SSMs)—specifically, the Mamba framework—for highly efficient, scalable video and vision processing. Key advances under the “MambaVF” name include lightweight, flow-free video fusion for multi-modal applications, end-to-end optical flow estimation, cross-layer fusion acceleration in vision Mamba models, and spatio-temporal modeling in video super-resolution. These systems are unified by their reliance on SSM layers inspired by the Mamba [Gu & Dao, 2024] design: a linear-time sequence modeling mechanism that substitutes for or enhances Transformer-based attention in long sequence or high-dimensional settings.

1. Foundations: State Space Models and the Mamba Architecture

Mamba is an instantiation of the classical linear state-space model (SSM), extended and parameterized to enable competitive accuracy in domain-general vision and sequential tasks:

• Each SSM layer maintains a hidden state $s_t \in \mathbb{R}^{d_\text{state}}$ updated recursively via

$$s_t = A\,s_{t-1} + B\,x_t,\quad y_t = C\,s_t + D\,x_t$$

where $A, B, C, D$ are learnable projections with possible additional structure (e.g., diagonal, convolutional, low-rank).

• The architecture supports parameterized state persistence and local mixing through convolution and expansion factors, yielding practical variations such as

$$s_t = \mathrm{diag}(\boldsymbol{\alpha})\,s_{t-1} + W_{\mathrm{in}}\,x_t$$

and output projection by concatenation followed by $W_{\mathrm{out}}$.

• Stacked Mamba blocks can be composed with residual connections and support linear overall time and memory complexity in input length $T$ and feature dimension $d$.

This SSM paradigm underpins the diverse MambaVF architectures, providing a scalable alternative to the quadratic complexity of self-attention as found in standard Transformers (Ying et al., 4 Jun 2025, Zhao et al., 5 Feb 2026, Du et al., 10 Mar 2025).

2. MambaVF for Video Fusion: Architecture and Methodology

The “MambaVF: State Space Model for Efficient Video Fusion” framework (Zhao et al., 5 Feb 2026) reconceptualizes video fusion as a sequential state-update across space and time, thereby removing the dependency on costly explicit motion estimation:

• Pipeline Overview:

1. Patch (Tubelet) Embedding: Each source stream (e.g. multi-exposure or multi-focus video) is embedded into a grid via 3D patchification.
2. Dual-Stream Tri-Axis Mamba Encoder: Each stream is processed by a stack of Vision State-Space (VSS) blocks; each block applies LayerNorm, spatio-temporal bidirectional (STB) SSM scanning, and a residual connection.
3. Fusion and Decoding: Features are concatenated and passed through further VSS blocks and residual 2D decoders to reconstruct fused frames.

• Spatio-Temporal Bidirectional (STB) SSM Module:
  • The core fusion occurs by repeatedly scanning the feature tensor along eight spatio-temporal trajectories: spatial row- and column-major order, pixel-wise temporal (across fixed locations), and their reversals.
  • For each trajectory, a hidden state is propagated using SSM recurrences:

  $$h_t = \bar{A} h_{t-1} + \bar{B} x_t;\quad y_t = C h_t + D x_t$$ - The output features from all directions are aggregated to produce the fused representation.

• Alignment Without Flow:

  • Unlike prior approaches (e.g. UniVF), the model dispenses with explicit flow estimation and warping, instead relying on implicit context propagation via the latent state, which is updated as the scan progresses.

The complexity per VSS block is $O(N d^2)$, where $N = T \cdot H \cdot W$, in contrast to $O(N^2 d)$ for self-attention, supporting practical scaling to long sequences and high resolution (Zhao et al., 5 Feb 2026).

3. Comparative Performance and Computational Analysis

The efficiency gains of MambaVF models are significant across fusion, retrieval, and flow estimation tasks:

Method	Params (M)	FLOPs (G)	Latency (ms, GH200)
TemCoCo	19.21	147.05	18.3
UniVF	9.16	78.23	71.1
MambaVF	0.71	8.77	33.4

• Parameter/FLOP Reduction: MambaVF achieves up to 92.25% reduction in parameters and 88.79% in FLOPs compared to flow-based UniVF, with latency improved by 2.1× in joint multi-task training scenarios.
• Quality Metrics: Across multi-exposure, multi-focus, infrared-visible, and medical fusion, MambaVF matches or outperforms state-of-the-art models in spatial quality (e.g. VIF, SSIM, MI, $Q^{AB/F}$) and temporal stability (BiSWE, MS2R).
• Robustness and Failure Modes: The model is robust to common fusion challenges but may degrade under extreme occlusions or minimal sequence length due to its scan pattern rigidity (Zhao et al., 5 Feb 2026).

4. Extensions: MambaVF in Optical Flow, Super-Resolution, and Efficient Vision

The “MambaVF” designation also encompasses the following advancements:

a. End-to-End Optical Flow Estimation

MambaVF (also called MambaFlow (Du et al., 10 Mar 2025)) deploys SSMs for both intra-frame (Self-Mamba) and inter-frame (Cross-Mamba) dependency modeling, integrated with a global-matching cost volume and a flow propagation module (PulseMamba):

• PolyMamba blocks substitute Transformer attention, achieving linear $O(ND)$ scaling.
• On the Sintel benchmark, MambaFlow attained EPE=1.60 (Clean), outperforming or matching prior methods and running 18% faster than GMFlow (Du et al., 10 Mar 2025).

b. Vision Mamba Acceleration via Cross-Layer Fusion

Fast Vision Mamba (Famba-V, referenced as MambaVF (Shen et al., 2024)) augments Vision Mamba (Vim) by dynamically fusing redundant tokens across selected layers:

• Token fusion is adaptive, using cosine similarity to select highly similar token pairs for merging in upper or interleaved layers.
• Memory and training time reductions of up to 40% and 15% are demonstrated with negligible accuracy loss on CIFAR-100 (Shen et al., 2024).

c. Video Super-Resolution

VSRM introduces dual Mamba block alternation—spatial-to-temporal (S2T-Mamba) and temporal-to-spatial (T2S-Mamba)—to aggregate long-range dependencies:

• Deformable Cross-Mamba alignment is employed for dynamic frame alignment, and a frequency-domain Charbonnier-like loss improves high-frequency texture retention.
• Empirical results show state-of-the-art PSNR/SSIM across REDS4, Vimeo-90K-T, and Vid4, and superior effective receptive fields compared to windowed attention or CNN approaches (Tran et al., 28 Jun 2025).

5. Applications and Generalization

MambaVF is deployed across a diverse set of tasks:

• Video Fusion: Multi-exposure, multi-focus, infrared-visible, and medical video fusion, with unified architecture and significant computational savings (Zhao et al., 5 Feb 2026).
• Optical Flow Estimation: Dense correspondence prediction suitable for motion analysis at high efficiency (Du et al., 10 Mar 2025).
• Video Retrieval: Multi-Mamba with temporal fusion enables effective partially relevant video retrieval by aligning long sequences in linear time (Ying et al., 4 Jun 2025).
• Vision-Language Integration and Generalized Segmentation: Mamba-based adapters (MFuser/MVFuser) bridge frozen vision foundation and vision-LLMs, supporting domain-generalized segmentation at linear scaling (Zhang et al., 4 Apr 2025).
• Super-Resolution: VSRM's Mamba backbone expands global receptive fields while maintaining latency and parameter efficiency (Tran et al., 28 Jun 2025).

6. Limitations and Future Directions

While MambaVF models have established new operating points for efficiency, scalability, and accuracy, some limitations are recognized:

• Fixed Scan Patterns: Performance may be suboptimal in highly irregular motion or occlusion, motivating exploration of learnable scan order or dynamic routing (Zhao et al., 5 Feb 2026).
• Fusion Granularity: Excessive early token merging degrades accuracy in cross-layer fusion; adaptive strategies may improve trade-offs (Shen et al., 2024).
• Coarse Moment Boundaries: Retrieval applications are limited by lack of precise temporal localization, suggesting potential gains via end-to-end training or integration with large vision-LLMs (Ying et al., 4 Jun 2025).
• Generalizability to Unseen Distributions: While implicit alignment methods are robust, catastrophic failure may occur under large occlusions or out-of-distribution shifts.

Proposed future research includes adaptive scan priorities for SSM, integration with event-based sensors, broad application to denoising or HDR reconstruction, and embedding Mamba layers with end-to-end trainable encoders in vision transformers (Zhao et al., 5 Feb 2026, Ying et al., 4 Jun 2025, Du et al., 10 Mar 2025).

7. Summary Table of Major MambaVF Variants

Variant	Principal Task	Key Innovation	Core Complexity	Notable Metric / Result	Reference
MambaVF	Video Fusion	ST Bidirectional SSM, flow-free fusion	O(N d²)	2.1× faster, 92% fewer params	(Zhao et al., 5 Feb 2026)
MambaFlow	Optical Flow	PolyMamba (Self/Cross-SSM), PulseMamba	O(ND)	EPE=1.60 (Sintel Clean)	(Du et al., 10 Mar 2025)
Famba-V	Efficient Vision	Cross-layer token fusion in Vim	O(N'D²+N'^2 D)	-40% memory, -15% time (@top-1 acc)	(Shen et al., 2024)
MamFusion	Video Retrieval	Multi-Mamba with Temporal Fusion	O(Td)	State-of-the-art SumR/Recall@K	(Ying et al., 4 Jun 2025)
VSRM	Video Super-Resolution	Dual S2T/T2S Mamba, deformable cross-M	O(LN)	+0.25–0.4 dB PSNR over SOTA	(Tran et al., 28 Jun 2025)

MambaVF models, by exploiting the structured efficiency of state-space recurrences, have set new standards in scalable video and vision processing for both academic and real-world applications. Their linear or near-linear complexity and generality position SSM-based architectures as central to next-generation, resource-efficient sequence modeling pipelines.