MambaTrans: Hybrid SSM and Transformer Models

• MambaTrans is a family of architectures combining Mamba-style selective state space models with Transformer attention, enabling efficient linear-time sequence processing.
• Hybrid designs employ serial, interleaved, partial substitution, and dynamic switching to balance global reasoning and efficiency across diverse applications.
• Transfer learning and parameter-efficient adaptation strategies facilitate incorporating pretrained Transformer modules within MambaTrans frameworks.

Searching arXiv for papers on Mamba–Transformer hybrids and related “MambaTrans” uses. {"query":"Mamba Transformer hybrid MambaTrans arXiv", "max_results": 10} Searching arXiv for “TransMamba” and “MambaTrans” specifically. {"query":"TransMamba MambaTrans arXiv", "max_results": 10} MambaTrans is best understood as a family of architectures and adaptation strategies that couple Mamba-style selective state space models with Transformer components, or use Mamba as a targeted substitute for Transformer modules. The term is not used in a single canonical sense across the literature. A plausible synthesis is that it denotes designs in which Mamba contributes linear-time sequence modeling, recurrent state, or efficient token compression, while Transformers remain responsible for global relational modeling, cross-attention, or task-specific reasoning. In some papers this appears as an explicit hybrid backbone; in others it appears as a transfer-learning, PEFT, or architectural co-design principle for replacing or complementing attention (Shao et al., 2024, Li et al., 31 Mar 2025, Ham et al., 2024). The label also names a specific modality translator for infrared–visible image fusion, whose core block combines mask-image-text cross-attention with a 3D-Selective Scan Module (Xu et al., 11 Aug 2025).

1. Conceptual scope and formal basis

At the core of most MambaTrans formulations is the selective state space model. Several papers restate the continuous-time SSM as

$$h'(t) = A h(t) + B x(t), \qquad y(t) = C h(t) + D x(t),$$

or in a simplified form without the skip term, and then discretize it with zero-order hold to obtain recurrent updates over tokens or patches (Ham et al., 2024, Shao et al., 2024). This yields a token-sequential hidden state that is updated in linear time with respect to sequence length, in contrast to self-attention, whose time and memory scale quadratically in token count.

The resulting contrast with Transformers is the principal motivation for MambaTrans designs. Transformer self-attention directly computes global token-token interactions, but incurs the familiar $O(L^2)$ scaling. Mamba replaces that with a learned state evolution, sometimes augmented with input-dependent parameters $\mathbf{B}, \mathbf{C}, \Delta$, 1D convolutions, and gated projections, thereby preserving long-context processing while avoiding explicit all-pairs attention (Shao et al., 2024, Ma et al., 2024). In traffic forecasting, this is framed as the difference between attention’s lack of a unified summary state and an SSM’s explicit recurrent memory; in time-series foundation modeling, it is the reason a Mamba backbone can remain linear in tokenized patch length while Transformer-based time-series foundation models remain quadratic (Shao et al., 2024, Ma et al., 2024).

A second foundational theme is the growing claim that Transformer attention and Mamba-style SSMs are not wholly disjoint formalisms. ST-MambaSync rewrites the discrete SSM update into an attention-like term plus a residual state term, and explicitly interprets Mamba as “attention mechanism within a ResNet framework” (Shao et al., 2024). TransMamba goes further by unifying Transformer and Mamba through shared parameter matrices, identifying correspondences such as $$\mathcal{W}_{\mathbf{Q}} \equiv \mathcal{W}_{\mathbf{C}}$$, $$\mathcal{W}_{\mathbf{K}} \equiv \mathcal{W}_{\mathbf{B}}$$, and $\mathcal{W}_{\mathbf{V}} \equiv \mathcal{W}_{x}$, and then using a Memory Converter to map attention-side features into SSM-compatible states at switching points (Li et al., 31 Mar 2025). This suggests that MambaTrans is not merely a pragmatic hybridization recipe, but also a program for exploiting an increasingly explicit algebraic correspondence between attention kernels and selective scans.

2. Recurrent architectural patterns

A first recurring pattern is serial composition, in which Transformer layers precede Mamba layers. ST-MambaSync is a clean instance: a Spatial–Temporal Transformer first models temporal and spatial self-attention, producing $$\mathbf{Z}^{(sp)} \in \mathbb{R}^{T \times N \times d_h}$$, after which an ST-mixer reshapes the tensor into a 1D sequence of length $\mathcal{T}=T\times N$ and a selective SSM processes it in linear time before a regression head predicts future traffic (Shao et al., 2024). MaskMamba reaches a similar conclusion for masked image generation: among grouped-parallel and serial variants, the best-performing backbone is serial v2, with Mamba blocks in early depth and Transformer blocks in later depth, i.e. an “MMMM…SSSS” arrangement (Chen et al., 2024).

A second pattern is interleaving, where Mamba and Transformer blocks alternate or are periodically distributed through depth. MamTra, built from a pretrained autoregressive Transformer TTS model, explores BlockBeg and BlockEnd schedules and finds that interleaving Transformer and Mamba blocks is a viable way to retain global semantic structure while reducing inference VRAM usage by up to 34% (Nguyen et al., 12 Mar 2026). MaTVLM, a hybrid VLM, replaces 12.5%, 25%, or 50% of the Transformer decoder layers with Mamba-2 layers and reports that evenly distributed Mamba layers outperform clustering them at the beginning or middle, while putting them all at the end fails badly (Li et al., 17 Mar 2025).

A third pattern is partial within-layer substitution. DRAMA keeps Transformer cross-attention in the decoder, but replaces the query self-processing stage with Mamba in each Mamba-Transformer decoder block, and separately replaces Transformer-based multimodal fusion with a Mamba Fusion module over concatenated camera and LiDAR feature sequences (Yuan et al., 2024). Streaming ASR adopts the complementary pattern: a Mamba encoder handles high-rate acoustic frames, while a causal self-attention decoder operates on low-rate token embeddings produced by Unimodal Aggregation (Fang et al., 2024). These examples are important because they show that MambaTrans need not be defined at the whole-model level; it often emerges as a local substitution policy over whichever submodule is most sequence-length-sensitive.

A fourth pattern is dynamic switching. TransMamba formalizes “TransPoints” that divide a sequence inside a layer: attention is used on tokens before the TransPoint, Mamba on tokens after it, and a Memory Converter bridges the two regimes so that context is not discarded during the switch (Li et al., 31 Mar 2025). This is the most explicit realization of MambaTrans as a continuously variable architecture rather than a static mixture.

3. Interfaces between 2D, multimodal, and 1D sequence processing

One of the most persistent technical difficulties for Mamba-based multimodal systems is the conversion of structured visual or spatial data into a 1D sequence suitable for selective scan. Query-based Cross-Modal Projector Bolstering Mamba Multimodal LLM attacks this problem directly. Its Q-Mamba projector inserts a learnable query sequence between a SigLIP vision encoder and a pretrained Mamba LLM, using a unidirectional Mamba layer over the queries and cross-attention from queries to visual features. The result is a compressed vision-token sequence of configurable length—128, 256, 512, or 729 queries—that can be concatenated with text tokens for a Mamba LLM without manually designing a 2D scan order (Eom et al., 3 Jun 2026). The paper explicitly positions this as a Mamba-based analogue of BLIP-2’s Q-Former.

MaskMamba addresses the same interface problem in a different way. Because masked image modeling is bidirectional rather than causal, it redesigns Bi-Mamba into Bi-Mamba-v2 by replacing causal convolutions with standard convolutions and replacing multiplicative branch fusion with concatenation followed by projection. This yields a bidirectional, non-causal Mamba block specialized for masked prediction, and then composes it with Transformer layers in the upper half of the backbone (Chen et al., 2024). The resulting model uses in-context conditioning by concatenating class or text tokens directly into the image-token stream, rather than introducing explicit cross-attention.

The explicit MambaTrans paper in multimodal fusion image translation pushes the multimodal interface further by combining fused infrared–visible images, segmentation masks, and multimodal LLM-generated text descriptions. Its Multimodal State Space Block contains a Text-Vision State Space Module and a Multimodal Cross Attention module; its 3D-Selective Scan scans four diagonal image directions and one text sequence, then fuses them as

$y = y_1 + y_2 + y_3 + y_4 + y_{\text{text}}.$

Mask features then modulate the fused state-space output before reconstruction into a translated image that can be consumed by frozen visible-trained detectors and segmenters (Xu et al., 11 Aug 2025). This is a particularly literal realization of MambaTrans as a cross-modal translator rather than a generic sequence backbone.

A related but more domain-specific interface appears in DRAMA. Camera features and LiDAR BEV features are reshaped into token sequences, concatenated, processed by Mamba-2, and then split back into modality-specific spatial tensors. The decoder then uses a Mamba-Transformer block in which Mamba handles query self-processing and Transformer cross-attention aligns trajectory queries with encoded context (Yuan et al., 2024). The broader implication is that MambaTrans often appears precisely where a model must repeatedly alternate between structured arrays and long 1D sequences.

4. Transfer learning, initialization, and parameter-efficient adaptation

A major strand of the literature treats MambaTrans not as a fixed backbone but as a transfer mechanism between Transformer and Mamba regimes. Fast Universal Architecture Adaption from Transformers to Mamba introduces a two-stage framework in which a pretrained Transformer acts as teacher and a Mamba architecture acts as student. Its main components are Weight Subcloning and Adaptive Bidirectional Distillation (WSAB), together with latent-space feature alignment and, for multimodal settings, a cross-Mamba module that injects language awareness into visual Mamba features (Chen et al., 21 Feb 2025). This framework is demonstrated on image classification, VQA, and text–video retrieval, and is explicitly motivated by the desire to reuse publicly available Transformer checkpoints such as DeiT, CLIP, and LLaVA rather than training Mamba models from scratch.

Flexibly Switching between Transformer and Mamba goes one step further by treating Transformer and Mamba as two modes of the same parameterized layer. Its Memory Converter computes an SSM-compatible state from attention-side features at a TransPoint, and its schedules explore layer-specific, broad-range, and fine-grained switching policies. The best schedule uses a repeated pattern of TransPoints

$[0, 128, 256, 512, 1024, 2048, 4096, 8192],$

which the paper argues yields a smoother transition than layer-shared schedules (Li et al., 31 Mar 2025). This is the clearest formulation of MambaTrans as a runtime-selectable architecture.

MaTVLM adopts a narrower but practically important transfer strategy. It substitutes a portion of a pretrained VLM decoder with Mamba-2 layers, initializes the Mamba-2 projections from the corresponding attention weights using the linearized attention–SSM mapping, and then trains only the Mamba-2 layers and the connector with a single-stage distillation loss

$O(L^2)$0

with $O(L^2)$1, $O(L^2)$2, and $O(L^2)$3 in the final configuration (Li et al., 17 Mar 2025). MamTra uses a closely related strategy in TTS: selected attention blocks are replaced by Mamba blocks, Mamba is initialized from the teacher’s Q/K/V weights, and the hybrid is distilled using cross-entropy, logit, and embedding-level losses (Nguyen et al., 12 Mar 2026).

Parameter-efficient adaptation adds another dimension. Parameter Efficient Mamba Tuning via Projector-targeted Diagonal-centric Linear Transformation argues that, in Mamba, projectors rather than SSMs are the primary carriers of transferable knowledge. It proposes ProDiaL,

$O(L^2)$4

which freezes the original projector $O(L^2)$5, learns a diagonal-centric block-diagonal transform $O(L^2)$6, a scaling vector $O(L^2)$7, and a low-rank off-diagonal correction. The broader lesson, stated explicitly for “MambaTrans”-like designs, is that projector-centric PEFT is the natural adaptation surface for Mamba blocks, unlike the attention-centric PEFT conventions inherited from Transformers (Ham et al., 2024).

5. Representative instantiations and empirical results

The literature now spans a wide range of MambaTrans instantiations, and the empirical evidence is domain-specific rather than uniform.

In traffic forecasting, ST-MambaSync couples a single ST-Transformer layer with a single ST-Mamba layer and reports the best accuracy–efficiency trade-off on PEMS08, where the 1 Mamba + 1 attention hybrid achieves MAE 13.30, RMSE 23.144, and MAPE 8.80, outperforming both a pure 1-layer attention model and a pure 1-layer Mamba model on MAE and MAPE (Shao et al., 2024). The same paper shows that deeper hybridization is not automatically beneficial: the 2 Mamba + 1 attention setting yields unexpectedly high MAPE 10.98 on PEMS08.

In image generation, MaskMamba reports that its redesigned Bi-Mamba-v2 yields a 17.77% inference speedup over the original Bi-Mamba block, and that the full hybrid MaskMamba-L achieves FID 6.61 and IS 127.74 on class-conditional ImageNet 256×256, improving over both Bi-Mamba-v2-L and Transformer-L at similar parameter scales (Chen et al., 2024). At very high resolution, its block-level benchmark at 2048×2048 reports that Bi-Mamba-v2 is 1.5× faster than Transformer, corresponding to a 54.44% speed improvement (Chen et al., 2024).

In vision-language modeling, MaTVLM preserves most of its Transformer teacher’s benchmark performance while achieving up to 3.6× faster inference and reducing GPU memory consumption by 27.5% (Li et al., 17 Mar 2025). Query-based Cross-Modal Projector Bolstering Mamba Multimodal LLM shows a different efficiency–accuracy trade-off: with 256 queries, Q-Mamba improves training and inference time over Cobra and VL-Mamba while remaining competitive across VQA-v2, GQA, TextVQA, POPE, and MMBench; with 729 queries, it reaches the strongest overall accuracy among its own variants (Eom et al., 3 Jun 2026).

In speech, MamTra shows that a hybrid Mamba–Transformer TTS backbone can reduce inference VRAM usage by up to 34% without compromising speech fidelity, even though it is trained on only 0.5k hours, described as about 2% of the teacher’s original training data (Nguyen et al., 12 Mar 2026). In streaming ASR, a Mamba encoder combined with Unimodal Aggregation and optional lookahead reaches CER 5.55% with Avg latency 494 ms on AISHELL-1 when paired with 256 ms lookahead and Early Termination, and CER 6.08% with Avg latency 764 ms on AISHELL-2 with 448 ms lookahead (Fang et al., 2024).

In autonomous driving, DRAMA improves the NAVSIM baseline Transfuser from PDMS 0.835 to 0.855 while reducing parameters from 56 MB to 50.6 MB in the full configuration; its Mamba Fusion module alone raises PDMS to 0.848 and lowers parameters to 49.9 MB (Yuan et al., 2024). In imitation learning, Mamba as a motion encoder reports higher success rates than Transformer baselines despite higher RMSE, reaching 20/20 on cup placing and 18/20 on case loading for Mamba $O(L^2)$8, versus 16/20 and 13/20 for Transformer $O(L^2)$9 (Tsuji, 2024).

In time series, TSMamba uses forward and backward Mamba encoders plus compressed channel-wise attention and reports zero-shot performance comparable to Time-MoE, Moirai, TimesFM, Moment, and Chronos despite using significantly less training data, while also achieving strong full-shot performance, including average MSE 0.222 on Weather and 1.255 on ILI (Ma et al., 2024). In 3D medical segmentation, UlikeMamba_3d surpasses UlikeTrans_SRA with average Dice 87.45 versus 85.97, and the final UlikeMamba_3dMT reaches Dice 89.95 on AMOS and 90.60 on BraTS while remaining more efficient than the Transformer baselines considered (Wang et al., 25 Mar 2025). In crack segmentation, a VMamba-based encoder–decoder uses 15.6%–74.5% fewer parameters than representative CNN-based models, obtains up to 2.8% higher mDS, and processes high-resolution image input with up to 90.6% lower floating-point operations (Chen et al., 2024).

6. Misconceptions, unresolved issues, and future directions

A common misconception is that MambaTrans simply means “replace attention with Mamba everywhere.” The literature does not support that simplification. Several papers instead argue for complementarity. ST-MambaSync explicitly characterizes Transformer and Mamba as complements, not substitutes, and shows that deeper hybrid stacks can hurt rather than help (Shao et al., 2024). MamTra similarly finds that aggressive Mamba ratios such as 1:11 degrade intelligibility, even though perceptual metrics remain strong (Nguyen et al., 12 Mar 2026). MaTVLM shows that a 25% Mamba-2 ratio is stronger than 50%, reinforcing the point that hybrid depth allocation is a first-order design variable (Li et al., 17 Mar 2025).

A second misconception is that the SSM core is automatically the best transfer-learning target in Mamba. ProDiaL challenges exactly this assumption, reporting that projector-only tuning surpasses SSM-only tuning and that a near-identity diagonal-centric transform captures most task adaptation (Ham et al., 2024). For MambaTrans-like systems, this implies that replacing or adapting Mamba blocks should not be equated with editing the SSM dynamics alone; projector design and projector PEFT are central.

Another open question concerns scan order and directional structure. In 3D medical segmentation, complex scanning is not uniformly necessary: simpler methods often suffice, though Tri-scan helps in the most challenging settings (Wang et al., 25 Mar 2025). In multimodal Mamba LLMs, Q-Mamba seeks to avoid manual 2D-to-1D scan design altogether by letting queries compress visual tokens through cross-attention (Eom et al., 3 Jun 2026). These lines of work suggest that “how to linearize structure for Mamba” remains unsettled and highly task-dependent.

Pure Mamba limitations remain visible across domains. Several papers note weaker global context handling, unstable contextual learning, forgetting over long multimodal sequences, or weaker multitask generalization relative to Transformers (Li et al., 31 Mar 2025, Eom et al., 3 Jun 2026, Li et al., 17 Mar 2025). Conversely, Transformer-derived hybrids continue to inherit attention’s computational burden in the layers that remain attention-based. This suggests that the long-term research agenda for MambaTrans is less about choosing a winner than about refining the interface: dynamic switching schedules, better Memory Converters, stronger off-diagonal modeling in PEFT, query-based multimodal compression, and architectural co-design in which Mamba is assigned long-sequence stateful modeling while attention is reserved for the few operations that still benefit from explicit token-token interaction (Li et al., 31 Mar 2025, Ham et al., 2024, Eom et al., 3 Jun 2026).

A plausible overall conclusion is that MambaTrans has become a generic design language for sequence models that seek Transformer-level expressivity at lower asymptotic cost. The dominant lesson of the current literature is not that Mamba obsoletes Transformers, but that carefully staged combinations—serial, interleaved, projector-centric, or dynamically switchable—now form a coherent architectural regime in their own right (Shao et al., 2024, Li et al., 31 Mar 2025, Li et al., 17 Mar 2025).