---
title: Bidirectional Mamba Backbone
url: https://www.emergentmind.com/topics/bidirectional-mamba-backbone
type: topic
---

# Bidirectional Mamba Backbone

A Bidirectional Mamba Backbone is a neural architecture that generalizes the unidirectional state-space Mamba block into a two-way, parallel selective scan framework. This design enables efficient linear-time modeling of both forward and backward dependencies in sequential or spatial data while maintaining—or substantially improving—the scalability, expressiveness, and hardware efficiency relative to transformer-based self-attention. Initially developed to address the representational limitations imposed by Mamba’s original causal (unidirectional) recurrence, the bidirectional Mamba backbone has become a central component in state-of-the-art vision, sequence, and multimodal learning systems [2502.07161].

## 1. Algorithmic Structure of the Bidirectional Mamba Block

A prototypical bidirectional Mamba block processes a 3D token tensor \(T_{l-1}\in\mathbb{R}^{B\times M\times D}\), where \(B\) is batch size, \(M\) sequence (or patch) length, and \(D\) feature dimension. The main algorithmic steps are as follows:

1. **Layer Normalization**: Normalize the input,
   \[
   T'_{l-1} = \mathrm{Norm}(T_{l-1})
   \]
2. **Linear Projections**: Project tokens into latent spaces for the SSM and gating,
   \[
   x = \mathrm{Linear}^x(T'_{l-1}) \ ;\ z = \mathrm{Linear}^z(T'_{l-1})
   \]
3. **Directional SSM Pathways**:
   - For each direction (forward, backward):
     - Apply a direction-aware 1D conv + SiLU activation,
       \[
       x_o' = \mathrm{SiLU}(\mathrm{Conv1d}_o(x))
       \]
     - Produce SSM parameters (\(B_o, C_o\)), step-size \(\Delta_o=\log[1+\exp(\cdot)]\), and per-timestep SSM matrices:
       \[
       \bar{A}_o = \Delta_o \otimes \mathrm{Param}^A_o,\quad\bar{B}_o = \Delta_o \otimes B_o
       \]
     - Run a unidirectional SSM filter per pathway,
       \[
       y_o = \mathrm{SSM}(\bar{A}_o, \bar{B}_o, C_o)(x_o')
       \]
4. **Directional Output Gating**: Both \(y_\mathrm{fwd}\) and \(y_\mathrm{bwd}\) are gated by \(\mathrm{SiLU}(z)\),
   \[
   y_{\text{fwd}} \leftarrow y_{\text{fwd}} \odot \mathrm{SiLU}(z),\quad y_{\text{bwd}} \leftarrow y_{\text{bwd}} \odot \mathrm{SiLU}(z)
   \]
5. **Fusion and Output Projection**: The outputs are added and projected back to match \(D\),
   \[
   T_{l} = \mathrm{Linear}^T(y_{\text{fwd}} + y_{\text{bwd}}) + T_{l-1}
   \]
This block is recurrently stacked to build the full backbone [2502.07161].

## 2. Bidirectional State Space Model Formalism

Each directional SSM models the input sequence via a selective, input-dependent state transition:
\[
\dot{x}(t) = A\, x(t) + B\, u(t),\quad y(t) = C\, x(t)
\]
For discretized sequence positions:
\[
h_k = \overline{A}_k\, h_{k-1} + \overline{B}_k\, u_k,\quad y_k = C_k\, h_k
\]
where \(\overline{A},\overline{B},C\) are dynamically parameterized by small neural networks, enabling adaptive, hardware-friendly filtering. Bidirectional blocks independently apply this procedure in both time directions: forward (\(k=1,\ldots,M\)) and backward (\(k=M,\ldots,1\)), using direction-specific convolution, SSM parameters, and gating [2502.07161].

## 3. Fusion Mechanisms and Gating

A central aspect of bidirectional Mamba is the gating and fusion of forward and backward SSM streams. Each output is modulated by a shared gate vector (often derived from the input or from an additional projection). In the canonical construction:
\[
y_{\mathrm{mix}} = y_\mathrm{fwd}\odot\mathrm{SiLU}(z) + y_\mathrm{bwd}\odot\mathrm{SiLU}(z)
\]
This joint stream is then post-processed (typically by a linear projection plus residual shortcut). While the main survey [2502.07161] does not detail cross-scan interactions beyond this dual-gating, other variants in the literature may employ learnable gates, dense attention-fusion, or more sophisticated selection mechanisms for context-dependent prioritization between directions.

## 4. Empirical Impact and Architectural Advantages

Bidirectional Mamba blocks deliver several critical benefits over unidirectional SSMs or quadratic-complexity self-attention:
- **Global context restoration**: By fusing causal and anti-causal information, each position receives global sequence context, crucial for tasks such as dense prediction, localization, and semantic understanding.
- **Linear complexity**: The total computational and memory cost per block scales as \(O(B\,M(E\,N+\text{conv}+D))\), which is linear in sequence/image size \(M\), enabling practical deployment at resolutions (or sequence lengths) where transformers become infeasible.
- **Empirical superiority**: Benchmarks across vision, language, and multimodal domains demonstrate that bidirectional Mamba backbones achieve accuracy, mAP, or mIoU competitive with or outperforming strong transformer baselines, while using a fraction of the computational resources [2502.07161].

## 5. Architectural Variants and Open Design Axes

Variants of the bidirectional Mamba backbone have appeared under numerous model labels (Mamba, VMamba, VideoMamba, and others). All share the essential dual-scan selective SSM block with gating and residual integration, but may instantiate distinct choices for:
- **Direction-aware preprocessing**: Different convolutional kernels, position embeddings, or normalization per scan direction.
- **Depth and hierarchy**: Number of bidirectional layers, interleaving with downsampling, or hybrid stacking with attention blocks.
- **Parameter sharing**: Whether the forward and backward SSMs share parameters or maintain distinct independent kernels.
- **Selective scanning and cross-scan mechanisms**: While the baseline implementation uses only dual-gated fusion, extended architectures may experiment with cross-attention between directional streams or additional selective gating, but these are not specified in [2502.07161].

## 6. Scope and Limitations

The architectural description here, drawn from [2502.07161], covers in precise detail the structure and algorithmic flow of the “Vim Block Process 2” bidirectional SSM block: specifically, two direction-aware SSM pathways operating in parallel, gated and fused via elementwise operations and projected back to the input feature dimension. Other potential enhancements such as explicit cross-scan modules, advanced positional encoding schemes, hierarchical multi-stage designs, and full complexity or memory analysis are not detailed in the survey’s excerpt.

## 7. Summary Table: Bidirectional Mamba Block Workflow

| Step                | Operation/Equation                                                       | Output Shape    |
|---------------------|-------------------------------------------------------------------------|-----------------|
| Input               | \(T_{l-1}\)                                                             | \(B\times M\times D\) |
| LayerNorm           | \(T'_{l-1} = \mathrm{Norm}(T_{l-1})\)                                   | \(B\times M\times D\) |
| Projections         | \(x = \mathrm{Linear}^x(T'_{l-1})\), \(z = \mathrm{Linear}^z(T'_{l-1})\) | \(B\times M\times E\) |
| Conv1d+SiLU         | \(x_o' = \mathrm{SiLU}(\mathrm{Conv1d}_o(x))\) (fwd, bwd)               | \(B\times M\times E\) |
| SSM params          | \(B_o,\,C_o,\,\Delta_o\) via linear/softplus                            | \(B\times M\times N/E\) |
| SSM filtering       | \(y_o = \mathrm{SSM}(\bar A_o,\bar B_o,C_o)(x_o')\)                     | \(B\times M\times E\) |
| Output gating       | \(y_o \odot \mathrm{SiLU}(z)\)                                          | \(B\times M\times E\) |
| Fuse and project    | \(T_l = \mathrm{Linear}^T(y_\mathrm{fwd} + y_\mathrm{bwd}) + T_{l-1}\)  | \(B\times M\times D\) |

This table distills the layerwise computation for reproducibility and precise architectural understanding, allowing direct implementation of bidirectional SSM-based backbones for scalable, context-rich modeling in vision and related domains [2502.07161].

Source: https://www.emergentmind.com/topics/bidirectional-mamba-backbone