TransBiMamba: Transformer-Style Bidirectional Mamba

• TransBiMamba is a Transformer-style bidirectional Mamba encoder that replaces standard multi-head self-attention with a state-space-based mechanism for deepfake detection.
• It serves as a backbone variant within Fake-Mamba, improving modeling of temporal-channel dependencies and achieving competitive EER and min-tDCF benchmarks.
• The design illustrates a broader hybrid architecture combining Transformer structures with bidirectional Mamba mechanisms, inspiring related developments in various domains.

TransBiMamba denotes a Transformer-style bidirectional Mamba encoder whose clearest explicit formulation appears in Fake-Mamba, where multi-head self-attention in a standard Transformer is replaced by BiMamba for speech deepfake detection (Xuan et al., 12 Aug 2025). In that usage it is one of three backbone variants—TransBiMamba, ConBiMamba, and PN-BiMamba—inside a larger XLSR-based framework. This suggests that “TransBiMamba” is best understood in two layers: first, as the specific Transformer-derived BiMamba block family introduced in Fake-Mamba; second, as a broader informal label for hybrid Transformer–Mamba designs that combine self-attention-derived structure with bidirectional or otherwise hybrid state-space modeling in other application domains.

1. Definition and conceptual scope

In Fake-Mamba, TransBiMamba is defined succinctly: “We replace MHSA in the standard Transformer with BiMamba, termed TransBiMamba” (Xuan et al., 12 Aug 2025). The term “Trans” therefore refers to a Transformer-style encoder scaffold, while “BiMamba” refers to a bidirectional Mamba mechanism intended to replace the usual self-attention block. The model is not introduced as an autonomous standalone system unrelated to the rest of Fake-Mamba; rather, it is one candidate backbone inside a four-stage speech deepfake detection pipeline.

The immediate motivation is the paper’s critique of conventional Conformer- or Transformer-based speech deepfake detectors. The authors attribute two central issues to multi-head self-attention: quadratic complexity in sequence length, written as $\mathcal{O}(t^2)$, and a potential inability to capture joint temporal-channel dependencies that are important for localized synthetic-speech artifacts. Against that background, TransBiMamba represents the most direct attention-replacement variant in the paper’s design space: a standard Transformer form in which the MHSA submodule is swapped out for bidirectional state-space modeling.

The same paper explicitly distinguishes TransBiMamba from the two sibling variants. ConBiMamba applies the same replacement idea to a standard Conformer, whereas PN-BiMamba is a newly designed variant with Pre-LayerNorm stabilization and bidirectional feature fusion (Xuan et al., 12 Aug 2025). This matters because TransBiMamba is an important definitional baseline rather than the paper’s strongest-performing configuration.

2. Architectural role inside Fake-Mamba

Fake-Mamba is described as a four-stage framework consisting of frame-level feature extraction, backbone, utterance-level pooling, and classification head (Xuan et al., 12 Aug 2025). In the TransBiMamba instantiation, the front end is XLSR, a wav2vec 2.0-based cross-lingual self-supervised speech model. Given input audio, XLSR produces frame-level features

$S_f \in \mathbb{R}^{T \times C},$

which are then passed through a learnable linear projection to reduce channel dimensionality to

$S_f' \in \mathbb{R}^{T \times D}.$

The paper gives one concrete setting for this projection: 144 output dimensions.

The projected sequence $S_f'$ is then processed by a stack of TransBiMamba blocks. The resulting sequence is aggregated by Linear Attention Pooling into an utterance-level embedding

$S_u \in \mathbb{R}^{D},$

and a multi-layer perceptron classification head produces the binary real-versus-fake prediction (Xuan et al., 12 Aug 2025). The paper also states that XLSR is fine-tuned jointly with the backbone and classifier, with no layers frozen.

What is architecturally clear is the system-level feature flow: input waveform $\rightarrow$ XLSR $\rightarrow$ linear projection $\rightarrow$ TransBiMamba backbone $\rightarrow$ Linear Attention Pooling $\rightarrow$ MLP classifier. What is not fully specified is the internal block algebra of TransBiMamba itself. The paper says that “The structure of the TransBiMamba block is shown in Figure 1(a),” but does not textually enumerate the exact normalization order, residual structure, FFN placement, or the precise bidirectional fusion operator for this variant (Xuan et al., 12 Aug 2025). That asymmetry between conceptual clarity and block-level specificity is one of the defining characteristics of the literature on this term.

3. State-space formulation and the status of bidirectionality

The mathematical substrate used to justify TransBiMamba is the standard Mamba state-space formulation introduced in Fake-Mamba (Xuan et al., 12 Aug 2025). The paper gives the continuous-time SSM as

$S_f \in \mathbb{R}^{T \times C},$0

where $S_f \in \mathbb{R}^{T \times C},$1 is the input sequence at continuous time $S_f \in \mathbb{R}^{T \times C},$2, $S_f \in \mathbb{R}^{T \times C},$3 is the hidden state vector, $S_f \in \mathbb{R}^{T \times C},$4 is the output, and $S_f \in \mathbb{R}^{T \times C},$5, $S_f \in \mathbb{R}^{T \times C},$6, and $S_f \in \mathbb{R}^{T \times C},$7 are the state transition, input projection, and output projection terms. It then gives zero-order hold discretization through

$S_f \in \mathbb{R}^{T \times C},$8

$S_f \in \mathbb{R}^{T \times C},$9

followed by the discrete-time recurrence

$S_f' \in \mathbb{R}^{T \times D}.$0

The printed form of the second expression is exactly as given in the paper, although by context it is evidently the output projection step.

To motivate efficiency, the paper further writes the convolutional form

$S_f' \in \mathbb{R}^{T \times D}.$1

It then emphasizes the selective character of Mamba: $S_f' \in \mathbb{R}^{T \times D}.$2, $S_f' \in \mathbb{R}^{T \times D}.$3, $S_f' \in \mathbb{R}^{T \times D}.$4, and $S_f' \in \mathbb{R}^{T \times D}.$5 are dynamically updated based on the input $S_f' \in \mathbb{R}^{T \times D}.$6, making the mechanism input-selective and content-aware rather than a fixed LTI scan (Xuan et al., 12 Aug 2025).

The crucial nuance is that these equations explain the Mamba primitive, but not the TransBiMamba block specifically. For TransBiMamba, the paper does not provide a separate pair of forward/backward equations. The only explicit realization of bidirectional Mamba inside Fake-Mamba is given for PN-BiMamba, where the backward branch is written as

$S_f' \in \mathbb{R}^{T \times D}.$7

The associated textual explanation is that all three variants are BiMamba variants, so a forward Mamba branch models left-to-right dependencies, a backward Mamba branch models right-to-left dependencies, and the outputs are fused (Xuan et al., 12 Aug 2025). For TransBiMamba, however, that bidirectional construction remains an architectural intention rather than a fully exposed block derivation.

4. Empirical performance and training regime

TransBiMamba is evaluated in Fake-Mamba on ASVspoof 2021 LA, ASVspoof 2021 DF, and In-The-Wild (Xuan et al., 12 Aug 2025). Table 1 in the paper reports the following variant-specific results for TransBiMamba: on ASVspoof 2021 LA, EER $S_f' \in \mathbb{R}^{T \times D}.$8, 95% CI $S_f' \in \mathbb{R}^{T \times D}.$9, and min-tDCF $S_f'$0; on ASVspoof 2021 DF, EER $S_f'$1 and 95% CI $S_f'$2; on In-The-Wild, EER $S_f'$3 and 95% CI $S_f'$4. These results are competitive, but they do not define the best Fake-Mamba configuration.

The ranking among the three backbones is explicit. PN-BiMamba is best on all three benchmarks, while TransBiMamba is generally second-best and ConBiMamba is worst overall, although ConBiMamba is slightly better than TransBiMamba on 21DF by a small margin. The paper therefore uses TransBiMamba to demonstrate that replacing Transformer self-attention with bidirectional Mamba is viable, but also concludes that a more carefully redesigned BiMamba block outperforms the direct Transformer-style replacement.

The duration-wise analysis reinforces the same pattern. For TransBiMamba with $S_f'$5, In-The-Wild EER is 15.14 for utterances shorter than 3s, 9.51 for 3–4s, 7.26 for 4–5s, 7.22 for 5–6s, and 5.48 for longer than 6s. For TransBiMamba with $S_f'$6, the corresponding EERs are 10.99, 6.15, 4.19, 4.03, and 3.44. The paper comments that increasing the number of BiMamba blocks from 4 to 7 consistently lowers EER for all variants (Xuan et al., 12 Aug 2025).

The shared training configuration is also reported concretely. Audio is sampled at 16 kHz and fixed to 4.175 s during training, corresponding to 66,800 samples. Optimization uses Adam with initial learning rate $S_f'$7, weight decay $S_f'$8, and batch size 32. Training runs for up to 100 epochs, with early stopping if validation loss does not decrease for 7 consecutive iterations. The final model is obtained by averaging the weights from the top 5 epochs with the lowest EERs on the development set. Data augmentation uses RawBoost, with distinct settings for the 21LA and 21DF evaluations (Xuan et al., 12 Aug 2025).

5. Related architectural families and broader usage

Although the exact name “TransBiMamba” is explicitly instantiated in Fake-Mamba, multiple neighboring papers describe architectures that are close to the same design space. In 3D point cloud analysis, PointABM combines a Transformer front-end with a bidirectional Mamba/state-space back-end: point patches are formed by FPS and KNN, processed by a Transformer encoder, and then passed into a Mamba encoder composed of $S_f'$9 bidirectional Mambas (Chen et al., 2024). The paper is explicit that this is serial rather than parallel hybridization, and its preferred integration mechanism is residual fusion rather than concatenation.

In resting-state fMRI, BrainMT is described as “very close to what one would informally call a ‘TransBiMamba’ architecture”: a convolutional stem is followed by a bidirectional Mamba block with temporal-first scanning, and then by a Transformer block using multi-head self-attention to capture global dependencies (Kannan et al., 27 Jun 2025). The division of labor is explicit in the paper: Mamba models long-range temporal interactions efficiently, while the Transformer models global spatial relationships across the Mamba-enriched features.

In text-to-image diffusion, Dimba studies an interleaved Transformer–Mamba stack rather than a simple replacement strategy. Its repeated hybrid blocks combine attention, Mamba, cross-attention for text conditioning, and MLP layers, and the paper explicitly contrasts its bidirectional diffusion setting with causal mixed-language-model designs (Fei et al., 2024). In large language modeling, Hunyuan-TurboS represents the hybrid Transformer–Mamba part of the design space strongly, but not the bidirectional part: it is framed as a causal, deployment-oriented model using sparse attention, Mamba2, and MoE feed-forward layers (Team et al., 21 May 2025).

A different branch of the literature emphasizes switching or parameter sharing rather than explicit bidirectional encoders. TransMamba proposes a shared-parameter, sequence-level switching framework in which Transformer and Mamba share parameter matrices such as $S_u \in \mathbb{R}^{D},$0 and $S_u \in \mathbb{R}^{D},$1, and the model changes computational mode at a per-layer TransPoint (Li et al., 31 Mar 2025). In lightweight vision restoration, BMTNet supplies yet another close analogue: a parallel dual-branch Binarized Mamba-Transformer block that combines Bi-Mamba and Bi-Swin Transformer to capture global and local dependencies in parallel (Zhou et al., 20 Mar 2025). For multi-task dense prediction, BIM uses a Swin Transformer encoder and a Mamba decoder with Bidirectional Interaction Scan and Multi-Scale Scan, thereby implementing bidirectional task-order Mamba interaction rather than a direct Transformer-block replacement (Cao et al., 28 Aug 2025).

Taken together, these works indicate that the broader “TransBiMamba” design space includes at least four recurring patterns: Transformer-first then bidirectional Mamba, bidirectional Mamba then Transformer, interleaved Transformer–Mamba blocks, and parallel Transformer/Mamba branches. This suggests that the term functions more as a family resemblance across hybrid sequence models than as a single locked architectural grammar.

6. Terminological ambiguities, limitations, and interpretive cautions

The strongest limitation of TransBiMamba as a term is that it is not standardized across the broader Mamba literature. In Fake-Mamba, the meaning is precise but brief: replace MHSA in a standard Transformer with BiMamba (Xuan et al., 12 Aug 2025). Outside that paper, closely related systems often do not use the name at all, even when they are architecturally very close in spirit.

A second ambiguity concerns the prefix “Bi.” In Fake-Mamba, PointABM, BrainMT, and BIM, “Bi” refers to bidirectional Mamba or bidirectional interaction mechanisms. In “Bi-Mamba: Towards Accurate 1-Bit State Space Models,” however, “Bi” means binary or binarized rather than bidirectional (Tang et al., 2024). That paper is relevant to Mamba compression, but it does not define a bidirectional TransBiMamba-style architecture. Confusing these two uses can lead to category errors, particularly in survey writing.

A third caution is that not every Transformer–Mamba hybrid belongs naturally under the TransBiMamba label. TransMamba, for example, is centered on shared parameterization and hard switching between attention and SSM modes, rather than on a bidirectional Mamba replacement for Transformer self-attention (Li et al., 31 Mar 2025). Similarly, Hunyuan-TurboS is strongly hybrid but explicitly causal rather than bidirectional (Team et al., 21 May 2025). This suggests that “TransBiMamba” should be reserved for cases where both components of the compound are materially present: a Transformer-derived structural prior and a genuinely bidirectional Mamba mechanism.

Finally, Fake-Mamba itself leaves TransBiMamba partially underspecified. The paper gives a clear pipeline-level definition and solid benchmark evidence, but it does not provide full TransBiMamba block equations, TransBiMamba-specific parameter counts, FLOPs, or runtime, and it does not list TransBiMamba-specific hyperparameters distinct from the shared framework configuration (Xuan et al., 12 Aug 2025). The result is a term that is conceptually clear, empirically meaningful, and increasingly useful as an architectural descriptor, but still dependent on context for exact technical interpretation.