SecMamba Block in Mammo-Mamba

• SecMamba Block is a MambaVision-derived module that combines a selective state-space model with a SeqMoE gate for adaptive depth mixing.
• It is integrated into the deeper stages of Mammo-Mamba to improve long-range modeling and refine mammographic features.
• Empirical results demonstrate that its hybrid design with attention blocks significantly boosts accuracy and AUC in multi-view mammogram classification.

Searching arXiv for the cited SecMamba-related papers to ground the article. The SecMamba Block is a named module introduced in "Mammo-Mamba: A Hybrid State-Space and Transformer Architecture with Sequential Mixture of Experts for Multi-View Mammography" as a modified MambaVision mixer block augmented by a Sequential Mixture of Experts (SeqMoE) mechanism (Bayatmakou et al., 23 Jul 2025). In its explicit form, it combines a selective state-space model (SSM), a selective modulation gate, and a depth-wise expert gate that mixes the current block output with the previous block output. Within the literature summarized here, this is the principal source that uses the term SecMamba directly; the vision-Mamba survey instead catalogs related MambaVision-style blocks and mechanisms without introducing a module literally called SecMamba (Ibrahim et al., 11 Feb 2025).

1. Architectural role and placement

In Mammo-Mamba, the SecMamba block is embedded in a four-stage MambaVision backbone and is confined to the deeper stages, where high-resolution mammographic interpretation is treated as most dependent on adaptive long-range modeling (Bayatmakou et al., 23 Jul 2025). The stage allocation is explicit.

Stages 1–2 are frozen MambaVision convolutional stages, with no SecMamba and no SeqMoE. Stages 3–4 are modified to insert SecMamba + SeqMoE. In Stage 3, the architecture contains 10 “experts”: 5 SecMamba blocks followed by 5 self-attention blocks, with SeqMoE gating applied iteratively across the expert sequence. In Stage 4, a convolutional downsampling operation precedes 5 experts: 3 SecMamba blocks followed by 2 self-attention blocks, again with iterative SeqMoE gating (Bayatmakou et al., 23 Jul 2025).

Each SecMamba block operates on a token sequence

$$T_i^{(\ell)} \in \mathbb{R}^{N_P \times d_{\text{PE}}},$$

and returns an output of the same shape for the next expert. The model is also organized as a dual-stream system, with one stream for crop inputs and one for whole mammogram inputs; each stream contains its own copy of the relevant stages, so SecMamba functions independently in each stream (Bayatmakou et al., 23 Jul 2025).

This placement reflects the broader vision-Mamba design pattern in which selective SSM mixers are used as drop-in alternatives to attention in later, semantically richer processing stages (Ibrahim et al., 11 Feb 2025). In the specific Mammo-Mamba design, however, the distinctive contribution is not merely replacing attention with a Mamba block, but wrapping the MambaVision mixer in a depth-wise gating scheme that adaptively controls how much new computation is accepted at each expert depth (Bayatmakou et al., 23 Jul 2025).

2. Internal formulation of the SecMamba block

The paper describes the SecMamba block as a Mamba-style SSM block $\mathcal{B}_{\text{M}}$ combined with a SeqMoE gate. The SSM core follows a standard discrete state-space formulation (Bayatmakou et al., 23 Jul 2025):

$h_{i,k} = A h_{i,k-1} + B x_{i,k-1}$

$y_{i,k} = C h_{i,k}.$

Here, $x_{i,k} \in \mathbb{R}^{d_{\text{IE}}}$ is the input token at position $k$, $h_{i,k} \in \mathbb{R}^{d_{\text{HS}}}$ is the hidden state, and $y_{i,k} \in \mathbb{R}^{d_{\text{IE}}}$ is the observation. For efficiency, the model rewrites the SSM as a one-dimensional convolution along the token axis:

$$Y_i = [y_{i,1}, \dots, y_{i,k}, \dots, y_{i,N_P}]^T = K^T * X_i,$$

with kernel

$$K = C A^k B = \left[ C B, \, C A B, \dots, C A^{N_P-1} B \right].$$

The block begins from a token sequence

$\mathcal{B}_{\text{M}}$0

which is projected into an internal embedding space:

$\mathcal{B}_{\text{M}}$1

After the SSM convolution, a selective modulation gate is applied:

$\mathcal{B}_{\text{M}}$2

where $\mathcal{B}_{\text{M}}$3, $\mathcal{B}_{\text{M}}$4 is an element-wise sigmoid, and $\mathcal{B}_{\text{M}}$5 is element-wise multiplication. This gate is the content-dependent modulation step that makes the block selective in the Mamba sense.

The gated output is then projected back:

$\mathcal{B}_{\text{M}}$6

The paper writes the Mamba block output as

$\mathcal{B}_{\text{M}}$7

and equivalently in expanded form as

$\mathcal{B}_{\text{M}}$8

The notation is described as slightly noisy in the source synthesis, but the intended structure is clear: input projection $\mathcal{B}_{\text{M}}$9 SSM convolution $h_{i,k} = A h_{i,k-1} + B x_{i,k-1}$0 selective modulation $h_{i,k} = A h_{i,k-1} + B x_{i,k-1}$1 output projection $h_{i,k} = A h_{i,k-1} + B x_{i,k-1}$2 residual + LayerNorm (Bayatmakou et al., 23 Jul 2025). A plausible implication is that the implementation keeps the residual branch shape-compatible at this stage, either by aligning $h_{i,k} = A h_{i,k-1} + B x_{i,k-1}$3 and $h_{i,k} = A h_{i,k-1} + B x_{i,k-1}$4 or by using an implicit projection.

3. Sequential Mixture of Experts as depth-wise gating

What distinguishes SecMamba from a plain MambaVision mixer is the SeqMoE wrapper. In this design, the current block output is not simply added residually to the incoming representation; instead, it is interpolated with the previous SecMamba output through a learned gate (Bayatmakou et al., 23 Jul 2025).

Let the current Mamba output be $h_{i,k} = A h_{i,k-1} + B x_{i,k-1}$5 and the previous aggregated output be $h_{i,k} = A h_{i,k-1} + B x_{i,k-1}$6. The gating network is

$h_{i,k} = A h_{i,k-1} + B x_{i,k-1}$7

where $h_{i,k} = A h_{i,k-1} + B x_{i,k-1}$8 denotes mean pooling over the sequence dimension.

The final SecMamba output is then

$h_{i,k} = A h_{i,k-1} + B x_{i,k-1}$9

The paper characterizes this as “an adaptive residual connection controlled by $y_{i,k} = C h_{i,k}.$0, similar in spirit to a GRU-style gate, but without recurrence”, and as an MoE in the depth dimension, where the gate selects between different levels of computation (Bayatmakou et al., 23 Jul 2025). The experts are not channels or token subsets; they are entire blocks in depth, including both SecMamba blocks and self-attention blocks.

Several clarifications are important.

First, this is not sparse top-$y_{i,k} = C h_{i,k}.$1 routing. The formulation is a dense two-way mixture between the current expert and the accumulated previous representation. No top-$y_{i,k} = C h_{i,k}.$2 routing, load-balancing loss, or capacity constraint is described (Bayatmakou et al., 23 Jul 2025).

Second, although the prose refers to decisions “at each spatial/temporal position,” the published equation pools both tensors with $y_{i,k} = C h_{i,k}.$3 before gating. This means the mathematically specified gate is global per layer, not token-wise. The source synthesis therefore recommends following the equation for faithful implementation (Bayatmakou et al., 23 Jul 2025).

Third, the gating network is shared across all experts within a given stage, while different stages use different gating functions. Thus, Stage 3 has its own gate shared over its 10 experts, and Stage 4 has a separate gate shared over its 5 experts (Bayatmakou et al., 23 Jul 2025). This stage-local sharing preserves parameter efficiency while allowing stage-specific control.

4. Functional purpose in multi-view mammography

SecMamba was proposed for multi-view mammography, where the modeling problem combines very high-resolution image content, subtle lesion cues, and the need for long-range contextual interpretation across the breast (Bayatmakou et al., 23 Jul 2025). The paper’s stated motivation is that accurate and efficient interpretation of multi-view mammograms is essential for early detection, while Transformer-based multi-view classifiers suffer from quadratic complexity in the number of image patches.

Within this setting, the SecMamba block contributes in two distinct ways. The first is inherited from the MambaVision lineage: a selective SSM provides linear-time sequence modeling over patch tokens, replacing full pairwise attention while retaining a mechanism for content-dependent modulation (Ibrahim et al., 11 Feb 2025). The second is specific to Mammo-Mamba: content-adaptive depth control through SeqMoE allows the network to decide how strongly to trust a newly computed expert output versus the previously accumulated representation (Bayatmakou et al., 23 Jul 2025).

The paper interprets this as a form of content-adaptive feature refinement in deeper stages. The practical intuition offered is that easy cases may preserve more of the existing representation, whereas ambiguous cases may benefit from more aggressive refinement by new SecMamba or attention experts (Bayatmakou et al., 23 Jul 2025). Because the two-stream architecture separates cropped lesions from whole mammograms, this refinement can operate independently on local detail and broader anatomical context.

The expert sequence in Stages 3 and 4 also creates a hybrid computational regime. SecMamba experts contribute linear-time long-range modeling, while the smaller number of self-attention experts retains a stronger pairwise interaction mechanism where the architecture judges it most beneficial (Bayatmakou et al., 23 Jul 2025). This does not make SecMamba itself an attention-SSM hybrid block in the strict sense; rather, it places the SecMamba block inside a hybrid stage whose routing is controlled by SeqMoE.

5. Empirical behavior, efficiency, and training regime

The empirical results reported for Mammo-Mamba indicate that the SecMamba-plus-SeqMoE design contributes materially beyond the baseline dual-stream MambaVision configuration on CBIS-DDSM (Bayatmakou et al., 23 Jul 2025). The paper also states that the SSM part preserves the linear scaling of MambaVision mixers, while the gating network adds negligible overhead; the self-attention blocks remain $y_{i,k} = C h_{i,k}.$4, but they are limited in number.

The comparative results explicitly reported in the source synthesis are as follows.

Variant	Accuracy	AUC
MambaVision (whole only)	0.7383	0.7748
MambaVision (cropped only)	0.7517	0.7960
MambaVision Backbone + Dual-stream Experts	0.8322	0.8791
MambaVision Mixers 2-stream (no attention, only SecMamba experts)	0.8244	0.8631
Mammo-Mamba (SeqMoE + Dual-Stream Experts)	0.8792	0.9249

These numbers show two distinct effects (Bayatmakou et al., 23 Jul 2025). First, a dual-stream expertized backbone is markedly stronger than single-stream whole-only or cropped-only variants. Second, the full Mammo-Mamba configuration improves over the dual-stream MambaVision baseline. The source synthesis states this gain explicitly as approximately Accuracy: +0.047 and AUC: +0.046 when adding SeqMoE, that is, SecMamba blocks and gating, to the dual-stream setup (Bayatmakou et al., 23 Jul 2025).

The reported training configuration is shared across models rather than being unique to SecMamba, but it is relevant for reproducing the block in context. The optimizer is AdamW. The learning rate is $y_{i,k} = C h_{i,k}.$5 to $y_{i,k} = C h_{i,k}.$6. Weight decay is $y_{i,k} = C h_{i,k}.$7 to $y_{i,k} = C h_{i,k}.$8. Training lasts 100 epochs with a cosine annealing LR schedule. The batch size is 8. The loss is cross-entropy with label smoothing. Gradient clipping: max-norm 1.0 is used. The input resolution is 224 × 224, and the reported augmentation is random color jitter (Bayatmakou et al., 23 Jul 2025).

A common misconception is that the performance gain should be attributed solely to replacing attention with Mamba. The ablations do not support that reading. The MambaVision Mixers 2-stream variant, described as using only SecMamba experts and no attention, reports Accuracy 0.8244 and AUC 0.8631, which is below the MambaVision 2-stream baseline at Accuracy 0.8322 and AUC 0.8791 (Bayatmakou et al., 23 Jul 2025). The improvement therefore comes from the combined design—SecMamba, SeqMoE, self-attention, and dual-streaming—rather than from SecMamba in isolation.

6. Relation to the broader Mamba block design space

Within the broader Mamba literature, SecMamba is best understood as a MambaVision-derived mixer wrapped by a depth-wise expert gate, rather than as a new state-update law. This distinguishes it from several other Mamba developments that alter different parts of the design space.

In "A Survey on Mamba Architecture for Vision Applications," the central vision mechanisms include bidirectional scanning, multi-path scanning, Spatial and Channel Attention (SCAttn), Structure-Aware State Fusion (SASF), Direction Alternation Hi-Mamba Group (DA-HMG), and Hidden State Mixer SSD (HSM-SSD) (Ibrahim et al., 11 Feb 2025). SecMamba does not introduce a new scan topology of that kind; it inherits the MambaVision-style selective SSM core and changes the depth-wise integration mechanism instead.

"LBMamba: Locally Bi-directional Mamba" modifies the scan kernel by embedding a lightweight locally backward scan inside the forward selective scan and alternating scan directions across layers (Zhang et al., 19 Jun 2025). SecMamba does not perform this local bi-directional scan. "Block-Biased Mamba for Long-Range Sequence Processing" changes the selective SSM itself through block-wise selective dynamics and a channel-specific bias in $y_{i,k} = C h_{i,k}.$9 (Yu et al., 13 May 2025). SecMamba does not adopt that parameterization. "Mamba-3: Improved Sequence Modeling using State Space Principles" introduces exponential-trapezoidal discretization, a complex-valued state transition, and an optional MIMO SSM core (Lahoti et al., 16 Mar 2026). Again, SecMamba does not make that class of recurrence-level modification. "TransMamba: Flexibly Switching between Transformer and Mamba" unifies attention and Mamba through shared parameter matrices and TransPoint scheduling (Li et al., 31 Mar 2025), whereas SecMamba keeps a conventional stage-level separation between Mamba-style blocks and self-attention blocks. "Spectral Informed Mamba for Robust Point Cloud Processing" preserves the Mamba block and changes the ordering and traversal of tokens through Laplacian spectral methods (Bahri et al., 6 Mar 2025); SecMamba does not redefine token order.

This contrast matters because the name SecMamba can otherwise be misread as denoting a general Mamba variant with altered SSM internals. The evidence here suggests a narrower definition: SecMamba is primarily an architectural wrapper around a MambaVision mixer that adds SeqMoE-based adaptive depth mixing (Bayatmakou et al., 23 Jul 2025).

The main limitations reported or implied for SecMamba are correspondingly architectural rather than dynamical. The gating mixes only two experts at each step, namely the current expert and the previous aggregated representation. It uses global mean pooled summaries, not an explicitly token-wise routing rule. The paper describes no explicit MoE-specific regularization, such as load balancing, entropy regularization, routing temperature control, or capacity constraints (Bayatmakou et al., 23 Jul 2025). It also identifies broader application-level directions for future work, including missing views, variable breast density, limited tumor annotations, and multi-task setups such as joint calcification and mass classification (Bayatmakou et al., 23 Jul 2025).

Taken together, these characteristics place SecMamba in a specific niche within the Mamba ecosystem: it is a task-specialized, depth-routed, MambaVision-derived expert block for high-resolution mammographic sequence modeling, rather than a generic redesign of selective SSM dynamics.