---
title: Bidirectional Caduceus (BiMamba)
url: https://www.emergentmind.com/topics/bidirectional-caduceus-bimamba
type: topic
---

# Bidirectional Caduceus (BiMamba)

Bidirectional Caduceus (BiMamba) is a neural sequence modeling architecture that generalizes the Mamba Selective State-Space Model (SSM) to bidirectional contexts. By combining forward and backward SSM recurrences, BiMamba achieves efficient modeling of long-term dependencies across a broad spectrum of modalities—audio, speech, genomics, time series, and multi-dimensional data—while maintaining near-linear scaling in sequence length. Originally conceived as a competitive alternative to multi-head self-attention (MHSA), BiMamba integrates content-aware, globally receptive convolutional dynamics in both temporal directions, frequently augmented with spatiotemporal, frequency, or multi-dimensional extensions depending on the application.

## 1. Core Architecture and Mathematical Foundations

BiMamba generalizes the Mamba SSM block to process inputs bi-directionally by coupling a forward SSM (processing the sequence from left-to-right) with a backward SSM (processing right-to-left), fusing their outputs at each layer. The underlying mechanism is grounded in continuous-time linear state-space models parameterized as
\[
h'(t) = A\,h(t) + B\,x(t),\quad y(t) = C\,h(t) + D\,x(t),
\]
with zero-order hold discretization:
\[
\bar{A} = \exp(\Delta A), \quad \bar{B} = (\Delta A)^{-1}(\exp(\Delta A)-I)\Delta B,\quad h_k = \bar{A} h_{k-1} + \bar{B} x_k, \quad y_k = C^\top h_k.
\]
BiMamba instantiates two independent recurrences (forward and backward), operating as
\[
h^{f}_k = \bar{A} h^{f}_{k-1} + \bar{B} x_k,\qquad h^{b}_k = \bar{A} h^{b}_{k+1} + \bar{B} x_k,
\]
where sequence outputs are fused, typically via concatenation followed by a learned projection, averaging, or scalar mixing:
\[
z_k = W_o \bigl[y^f_k \oplus y^b_k\bigr],\qquad \text{or}\qquad z_k = \beta\, y^f_k + (1-\beta)\, y^b_k.
\]
Variants exist where the forward/backward SSMs share parameters for parameter efficiency, or operate with independent (external) projections for flexibility [2403.03234, 2405.12609, 2507.09570].

## 2. Complexity, Modeling Power, and Comparison to Self-Attention

BiMamba matches or surpasses self-attention in its capacity to capture long-range dependencies, but with critical computational advantages:
- **Computational cost:** BiMamba’s SSM layers scale as $\mathcal{O}(L D)$ with token length $L$ and hidden dimension $D$, compared to $\mathcal{O}(L^2 D)$ for MHSA [2405.12609, 2507.09570]. Efficient convolutional implementations (parallel scan or FFT) further accelerate inference.
- **Memory footprint:** The memory demand is linear in $L$, avoiding the quadratic scaling bottleneck of attention-based models, especially for long sequences or high-dimensional data [2403.03234, 2411.15380].
- **Global context:** Unlike strictly local convolutional nets, BiMamba SSMs compute global receptive fields via convolutional kernels dynamically synthesized from sequence content.
- **Bidirectional dependency:** Incorporating both past and future context is vital for non-autoregressive tasks such as speech recognition, sound event detection, and genomics. Bidirectional SSM yields empirically significant performance boosts over strictly causal (unidirectional) designs [2507.09570, 2405.20142, 2508.09294, 2405.12609].

## 3. Block Structure, Variants, and Engineering Extensions

Canonical BiMamba blocks instantiate the following pipeline:
1. **Preprocessing:** Layer normalization and linear projection of the input tensor to the desired hidden/state dimensionality (typical $D=512$ in audio, $N=16-256$ for SSM state).
2. **Bidirectional SSM:** Parallel forward-time and backward-time SSM passes as described above.
3. **Fusion:** Channel-wise concatenation plus projection, or scalar-weighted sum.
4. **Nonlinearity and Residual:** Application of batch or layer normalization, nonlinearities (ReLU or SiLU), and residual addition.
5. **Additional Modules:** Depending on the application, BiMamba often incorporates:
   - **Asymmetric convolutions:** Decomposition of mixing across time and frequency dimensions, e.g., 1D time (kernel $K_t$), 1D frequency (kernel $K_f$) [2507.09570, 2506.13455].
   - **Channel attention (ECA):** Per-channel reweighting for multimodal data [2405.20142].
   - **Dimension-agnostic convolutions:** For generalizing to 1D, 2D, or 3D data, e.g., via Nd-BiMamba2 [2411.15380].
   - **Gating:** Dynamic gating of SSM updates, especially in speech and sleep staging models.
   - **Specialized fusions:** E.g., scalar mixing (learned $\beta$) or sum, required for context-sensitive applications.
6. **Decoder/Head:** Task-specific linear or MLP heads for classification or regression.

A summary of architectural choices and their domain-specific instantiations appears below:

| Domain                  | Key Extension             | Reference         |
|-------------------------|--------------------------|-------------------|
| Stereo SELD             | Asymmetric conv (time/f) | [2507.09570]      |
| Genomics                | RC equivariant fusion     | [2403.03234]      |
| Sleep stage/PSG         | Channel attention (ECA)  | [2405.20142]      |
| 2D/3D data              | Dim-adaptive BiMamba2    | [2411.15380]      |
| Streaming ASR           | Trans-Chunk BiMamba      | [2602.11546]      |

## 4. Applications Across Modalities

BiMamba architectures have demonstrated empirical benefits across heterogeneous tasks:

- **Speech Recognition and Enhancement:** BiMamba, as a replacement for MHSA in Transformer/Conformer backbones, delivers lower word error rate (WER) and better speech enhancement metrics (PESQ, ESTOI) compared to unidirectional Mamba or attention models, with linear-time inference [2405.12609, 2602.11546]. The Trans-Chunk BiMamba further supports unified offline and streaming ASR with dynamic chunk sizing and full bidirectional context at every latency [2602.11546].
- **Sound Event Localization/Detection (SELD):** Replacement of Conformer decoders by BiMamba with asymmetric convolutions enables improved F1 scores ($F_{20^\circ}$) and reduced parameter counts for stereo SELD [2507.09570, 2506.13455].
- **Sleep Stage Classification:** In multichannel PSG, BiMamba with ECA achieves higher accuracy and F1 versus Transformer or RNN baselines, while using only a fraction of the parameters [2405.20142].
- **Trustworthy Speech Processing:** Fake-Mamba demonstrates that BiMamba can supplant attention in real-time speech deepfake detection, obtaining lower EER than advanced attention-based detectors at lower computational cost [2508.09294].
- **Genomics and DNA Modeling:** BiMamba, as the bidirectional module in the Caduceus framework, supports efficient long-range sequence modeling over kilobase- to megabase-scale DNA. The RC-equivariant extension (MambaDNA) enables equivariant sequence modeling for regulatory genomics and variant effect prediction, setting state-of-the-art on several biological sequence benchmarks despite competitive models being orders-of-magnitude larger [2403.03234].
- **Multi-Dimensional Data:** Nd-BiMamba2 applies the BiMamba concept to any spatial or spatiotemporal dimension (1D/2D/3D), supporting vision, volumetric, or timeseries tasks in a unified, portable, and efficient module [2411.15380].

## 5. Quantitative Performance and Efficiency Gains

Rigorous benchmarks consistently corroborate the advantages of BiMamba:

- **SELD (DCASE2025 Task 3):** BiMambaAC achieves $F_{20^\circ}=39.6\%$ (vs. Conformer’s $38.2\%$) using only 36% of the parameters (76M vs. 210M) and comparable MACs (4.63G vs. 4.69G) [2506.13455, 2507.09570].
- **Real-time Speech Deepfake Detection:** Fake-Mamba’s PN-BiMamba achieves EERs of 0.97%, 1.74%, and 5.85% on ASVspoof 21 LA, 21 DF, and In-the-Wild, outperforming attention-based XLSR-Conformer and XLSR-Mamba [2508.09294].
- **Speech Enhancement (LibriSpeech):** ExtBiMamba-6 reaches NB-PESQ 2.88 and ESTOI 76.69% (vs. Transformer-6’s 2.78 and 74.56%) [2405.12609].
- **Sleep Staging (ISRUC-S3):** CNN+ECA+1-BiMamba yields 0.852 accuracy and 0.824 F1 using just 0.47M parameters, outperforming DeepSleepNet and MixSleepNet [2405.20142].
- **Streaming ASR:** TC-BiMamba enables dynamic-latency streaming (offline Rescore WER 2.97%) at 1.3× training speedup, 50% less memory, equaling or surpassing fixed-chunk BiMamba and U2++ models at smaller model size [2602.11546].
- **Genomics (Regulatory Classification):** Caduceus-PS achieves 0.900 accuracy on Human Enhancer classification (Ensembl), outperforming long-range HyenaDNA (0.849) at <2M parameters [2403.03234].
- **Multi-Dimensional Data:** Nd-BiMamba2 delivers up to 8% increased accuracy in vision tasks with only a 2× FLOPs increase over unidirectional processing; adaptive padding strategies yield 10–30% compute savings [2411.15380].

## 6. Implementation, Training Practices, and Trade-Offs

All practical BiMamba instantiations employ a series of architectural and training best practices:
- **LayerNorm before block entry; residual connections after each block.**
- **Separate forward and backward linear projections for SSM input/output** (unless sharing is imposed for efficiency [2403.03234]).
- **Stacked block architectures, typically 4–7 layers deep for speech/audio, up to 16 for genomics.**
- **Small kernel asymmetric convolutions for time/frequency decoupling in audio.**
- **Adam optimizer, ReduceLROnPlateau scheduler, moderate to low learning rates (1e-3 to 3e-5), and batch sizes adjusted for hardware constraints.**
- **Projection weight sharing (in Caduceus) enables deeper BiMamba stacks at low parameter cost without accuracy loss [2403.03234].**
- **Dynamic chunk-size sampling in TC-BiMamba for universal streaming/offline ASR [2602.11546].**
- **ONNX and TorchScript compatibility for Nd-BiMamba2 ensures easy cross-hardware deployment [2411.15380].**

Bidirectionality in BiMamba roughly doubles compute per layer relative to unidirectional Mamba, but yields marked gains in downstream modeling tasks. Adaptive padding and modular interfaces mitigate memory/computation overheads in high-dimensional inputs [2411.15380]. Ablation studies uniformly find bidirectionality, gating, nonlinearity, and residuals as indispensable for state-of-the-art performance [2405.12609, 2507.09570, 2508.09294].

## 7. Limitations, Extensions, and Ongoing Developments

Limitations of BiMamba reported in the literature include:
- **2× computational overhead** versus unidirectional Mamba [2411.15380].
- **Simple fusion of forward/backward representations (sum or scalar mix);** richer cross-stream interactions are underexplored.
- **Parameter efficiency and representational capacity trade-off:** Although shared projections and weight-tying admit deeper networks, some applications may require flexibility from independent per-direction parameterization [2403.03234].
- **No explicit directional bias per spatial axis** in Nd-BiMamba2.
- **For semantic-oriented tasks, the FFN and residual block must be retained** to avoid degraded representational power [2405.12609].

Current and future research continues to:
- Investigate generalized fusion/attention across directions and tasks [2403.03234, 2411.15380].
- Extend BiMamba blocks to RC-equivariant, symmetry-preserving contexts (e.g., genomics) [2403.03234].
- Explore modular, dimension-agnostic block designs for universal deployment in high-dimensional data domains [2411.15380].
- Unify streaming and non-streaming processing without model bifurcation, as in Trans-Chunk BiMamba for ASR [2602.11546].
- Apply BiMamba to new domains including vision and time-series beyond speech and language [2508.09294, 2411.15380].

**References:**  
[2403.03234], [2405.12609], [2405.20142], [2411.15380], [2506.13455], [2507.09570], [2508.09294], [2602.11546].

Source: https://www.emergentmind.com/topics/bidirectional-caduceus-bimamba