---
title: SecMamba Block in Mammo-Mamba
url: https://www.emergentmind.com/topics/secmamba-block
type: topic
---

# SecMamba Block in Mammo-Mamba

Searching arXiv for the cited SecMamba-related papers to ground the article.
The **SecMamba Block** is a named module introduced in "Mammo-Mamba: A Hybrid State-Space and Transformer Architecture with Sequential Mixture of Experts for Multi-View Mammography" as a modified **MambaVision** mixer block augmented by a **Sequential Mixture of Experts (SeqMoE)** mechanism [2507.17662]. In its explicit form, it combines a **selective state-space model (SSM)**, a **selective modulation gate**, and a depth-wise expert gate that mixes the current block output with the previous block output. Within the literature summarized here, this is the principal source that uses the term *SecMamba* directly; the vision-Mamba survey instead catalogs related MambaVision-style blocks and mechanisms without introducing a module literally called SecMamba [2502.07161].

## 1. Architectural role and placement

In Mammo-Mamba, the SecMamba block is embedded in a four-stage MambaVision backbone and is confined to the deeper stages, where high-resolution mammographic interpretation is treated as most dependent on adaptive long-range modeling [2507.17662]. The stage allocation is explicit.

**Stages 1–2** are frozen MambaVision convolutional stages, with **no SecMamba** and **no SeqMoE**.  
**Stages 3–4** are modified to insert **SecMamba + SeqMoE**. In **Stage 3**, the architecture contains **10 “experts”**: **5 SecMamba blocks followed by 5 self-attention blocks**, with SeqMoE gating applied iteratively across the expert sequence. In **Stage 4**, a convolutional downsampling operation precedes **5 experts**: **3 SecMamba blocks followed by 2 self-attention blocks**, again with iterative SeqMoE gating [2507.17662].

Each SecMamba block operates on a token sequence
$$
T_i^{(\ell)} \in \mathbb{R}^{N_P \times d_{\text{PE}}},
$$
and returns an output of the same shape for the next expert. The model is also organized as a **dual-stream** system, with one stream for **crop** inputs and one for **whole mammogram** inputs; each stream contains its own copy of the relevant stages, so SecMamba functions independently in each stream [2507.17662].

This placement reflects the broader vision-Mamba design pattern in which selective SSM mixers are used as drop-in alternatives to attention in later, semantically richer processing stages [2502.07161]. In the specific Mammo-Mamba design, however, the distinctive contribution is not merely replacing attention with a Mamba block, but wrapping the MambaVision mixer in a depth-wise gating scheme that adaptively controls how much new computation is accepted at each expert depth [2507.17662].

## 2. Internal formulation of the SecMamba block

The paper describes the SecMamba block as a **Mamba-style SSM block** $\mathcal{B}_{\text{M}}$ combined with a **SeqMoE** gate. The SSM core follows a standard discrete state-space formulation [2507.17662]:
$$
h_{i,k} = A h_{i,k-1} + B x_{i,k-1}
$$
$$
y_{i,k} = C h_{i,k}.
$$

Here, $x_{i,k} \in \mathbb{R}^{d_{\text{IE}}}$ is the input token at position $k$, $h_{i,k} \in \mathbb{R}^{d_{\text{HS}}}$ is the hidden state, and $y_{i,k} \in \mathbb{R}^{d_{\text{IE}}}$ is the observation. For efficiency, the model rewrites the SSM as a one-dimensional convolution along the token axis:
$$
Y_i = [y_{i,1}, \dots, y_{i,k}, \dots, y_{i,N_P}]^T = K^T * X_i,
$$
with kernel
$$
K = C A^k B = \left[ C B, \, C A B, \dots, C A^{N_P-1} B \right].
$$

The block begins from a token sequence
$$
T_i \in \mathbb{R}^{N_P \times d_{\text{PE}}},
$$
which is projected into an internal embedding space:
$$
X_i = W_{\text{in}} T_i + b_{\text{in}},
\quad X_i \in \mathbb{R}^{N_P \times d_{\text{IE}}}.
$$

After the SSM convolution, a **selective modulation gate** is applied:
$$
Y_i^{\text{MD}} = G \odot Y_i, \quad G = \sigma(W_g X_i + b_g),
$$
where $G \in \mathbb{R}^{N_P \times d_{\text{IE}}}$, $\sigma(\cdot)$ is an element-wise sigmoid, and $\odot$ is element-wise multiplication. This gate is the content-dependent modulation step that makes the block selective in the Mamba sense.

The gated output is then projected back:
$$
Y_i^{\text{SM}} = W_{\text{out}} Y_i^{\text{MD}} + b_{\text{out}},
\quad Y_i^{\text{SM}} \in \mathbb{R}^{N_P \times d_{\text{PE}}}.
$$

The paper writes the Mamba block output as
$$
\mathcal{B}_{\text{M}}(X_i) = \text{LayerNorm}\Big( X_i + Y_i^{\text{SM}} \Big),
$$
and equivalently in expanded form as
$$
\mathcal{B}_{\text{M}}(X_i) = \text{LayerNorm} \Big( X_i + W_{\text{out}}\big[ (\sigma(W_g X_i + b_g) ) \odot ( K * X_i ) \big] + b_{\text{out}} \Big).
$$

The notation is described as slightly noisy in the source synthesis, but the intended structure is clear: **input projection $\rightarrow$ SSM convolution $\rightarrow$ selective modulation $\rightarrow$ output projection $\rightarrow$ residual + LayerNorm** [2507.17662]. A plausible implication is that the implementation keeps the residual branch shape-compatible at this stage, either by aligning $d_{\text{IE}}$ and $d_{\text{PE}}$ or by using an implicit projection.

## 3. Sequential Mixture of Experts as depth-wise gating

What distinguishes SecMamba from a plain MambaVision mixer is the **SeqMoE** wrapper. In this design, the current block output is not simply added residually to the incoming representation; instead, it is **interpolated** with the previous SecMamba output through a learned gate [2507.17662].

Let the current Mamba output be $\mathcal{B}_{\text{M}}(X_i)$ and the previous aggregated output be $\mathcal{B}_{\text{SecM}}(X_i^{(\ell-1)})$. The gating network is
$$
\begin{aligned}
\mathcal{G}\Big( &\mathcal{B}_{\text{M}}(X_i),\
\mathcal{B}_{\text{SecM}}(X_i^{(\ell-1)}) \Big) = \ &
\text{Softmax} \Big( W_{\text{out}}^{\mathcal{G}}\, \text{ReLU}\big( W_{\text{in}}^{\mathcal{G}} [ \mu(\mathcal{B}_{\text{M}}(X_i)),\
\mu(\mathcal{B}_{\text{SecM}}(X_i^{(\ell-1)})) ] \big) \Big),
\end{aligned}
$$
where $\mu(\cdot)$ denotes mean pooling over the sequence dimension.

The final SecMamba output is then
$$
\mathcal{B}_{\text{SecM}}(X_i)
= \mathcal{G}\,\mathcal{B}_{\text{M}}(X_i)
+ (1 - \mathcal{G})\,\mathcal{B}_{\text{SecM}}(X_i^{(\ell-1)}).
$$

The paper characterizes this as **“an adaptive residual connection controlled by $\mathcal{G}$, similar in spirit to a GRU-style gate, but without recurrence”**, and as an **MoE in the depth dimension**, where the gate selects between different levels of computation [2507.17662]. The experts are not channels or token subsets; they are **entire blocks in depth**, including both SecMamba blocks and self-attention blocks.

Several clarifications are important.

First, this is **not** sparse top-$k$ routing. The formulation is a **dense two-way mixture** between the current expert and the accumulated previous representation. No top-$k$ routing, load-balancing loss, or capacity constraint is described [2507.17662].

Second, although the prose refers to decisions “at each spatial/temporal position,” the published equation pools both tensors with $\mu(\cdot)$ before gating. This means the mathematically specified gate is **global per layer**, not token-wise. The source synthesis therefore recommends following the equation for faithful implementation [2507.17662].

Third, the gating network is **shared across all experts within a given stage**, while different stages use different gating functions. Thus, Stage 3 has its own gate shared over its 10 experts, and Stage 4 has a separate gate shared over its 5 experts [2507.17662]. This stage-local sharing preserves parameter efficiency while allowing stage-specific control.

## 4. Functional purpose in multi-view mammography

SecMamba was proposed for **multi-view mammography**, where the modeling problem combines very high-resolution image content, subtle lesion cues, and the need for long-range contextual interpretation across the breast [2507.17662]. The paper’s stated motivation is that accurate and efficient interpretation of multi-view mammograms is essential for early detection, while Transformer-based multi-view classifiers suffer from quadratic complexity in the number of image patches.

Within this setting, the SecMamba block contributes in two distinct ways. The first is inherited from the MambaVision lineage: a **selective SSM** provides linear-time sequence modeling over patch tokens, replacing full pairwise attention while retaining a mechanism for content-dependent modulation [2502.07161]. The second is specific to Mammo-Mamba: **content-adaptive depth control** through SeqMoE allows the network to decide how strongly to trust a newly computed expert output versus the previously accumulated representation [2507.17662].

The paper interprets this as a form of **content-adaptive feature refinement** in deeper stages. The practical intuition offered is that easy cases may preserve more of the existing representation, whereas ambiguous cases may benefit from more aggressive refinement by new SecMamba or attention experts [2507.17662]. Because the two-stream architecture separates **cropped lesions** from **whole mammograms**, this refinement can operate independently on local detail and broader anatomical context.

The expert sequence in Stages 3 and 4 also creates a hybrid computational regime. **SecMamba experts** contribute linear-time long-range modeling, while the smaller number of **self-attention experts** retains a stronger pairwise interaction mechanism where the architecture judges it most beneficial [2507.17662]. This does not make SecMamba itself an attention-SSM hybrid block in the strict sense; rather, it places the SecMamba block inside a hybrid stage whose routing is controlled by SeqMoE.

## 5. Empirical behavior, efficiency, and training regime

The empirical results reported for Mammo-Mamba indicate that the SecMamba-plus-SeqMoE design contributes materially beyond the baseline dual-stream MambaVision configuration on **CBIS-DDSM** [2507.17662]. The paper also states that the SSM part preserves the **linear scaling** of MambaVision mixers, while the gating network adds negligible overhead; the self-attention blocks remain **$\mathcal{O}(N_P^2)$**, but they are limited in number.

The comparative results explicitly reported in the source synthesis are as follows.

| Variant | Accuracy | AUC |
|---|---:|---:|
| MambaVision (whole only) | 0.7383 | 0.7748 |
| MambaVision (cropped only) | 0.7517 | 0.7960 |
| MambaVision Backbone + Dual-stream Experts | 0.8322 | 0.8791 |
| MambaVision Mixers 2-stream (no attention, only SecMamba experts) | 0.8244 | 0.8631 |
| Mammo-Mamba (SeqMoE + Dual-Stream Experts) | 0.8792 | 0.9249 |

These numbers show two distinct effects [2507.17662]. First, a dual-stream expertized backbone is markedly stronger than single-stream whole-only or cropped-only variants. Second, the full Mammo-Mamba configuration improves over the dual-stream MambaVision baseline. The source synthesis states this gain explicitly as approximately **Accuracy: +0.047** and **AUC: +0.046** when adding SeqMoE, that is, SecMamba blocks and gating, to the dual-stream setup [2507.17662].

The reported training configuration is shared across models rather than being unique to SecMamba, but it is relevant for reproducing the block in context. The optimizer is **AdamW**. The learning rate is **$5\times 10^{-5}$ to $7\times 10^{-5}$**. Weight decay is **$1\times 10^{-3}$ to $3\times 10^{-3}$**. Training lasts **100** epochs with a **cosine annealing LR schedule**. The batch size is **8**. The loss is **cross-entropy with label smoothing**. **Gradient clipping: max-norm 1.0** is used. The input resolution is **224 × 224**, and the reported augmentation is **random color jitter** [2507.17662].

A common misconception is that the performance gain should be attributed solely to replacing attention with Mamba. The ablations do not support that reading. The **MambaVision Mixers 2-stream** variant, described as using only SecMamba experts and no attention, reports **Accuracy 0.8244** and **AUC 0.8631**, which is below the **MambaVision 2-stream** baseline at **Accuracy 0.8322** and **AUC 0.8791** [2507.17662]. The improvement therefore comes from the combined design—**SecMamba, SeqMoE, self-attention, and dual-streaming**—rather than from SecMamba in isolation.

## 6. Relation to the broader Mamba block design space

Within the broader Mamba literature, SecMamba is best understood as a **MambaVision-derived mixer wrapped by a depth-wise expert gate**, rather than as a new state-update law. This distinguishes it from several other Mamba developments that alter different parts of the design space.

In "A Survey on Mamba Architecture for Vision Applications," the central vision mechanisms include **bidirectional scanning**, **multi-path scanning**, **Spatial and Channel Attention (SCAttn)**, **Structure-Aware State Fusion (SASF)**, **Direction Alternation Hi-Mamba Group (DA-HMG)**, and **Hidden State Mixer SSD (HSM-SSD)** [2502.07161]. SecMamba does not introduce a new scan topology of that kind; it inherits the MambaVision-style selective SSM core and changes the depth-wise integration mechanism instead.

"LBMamba: Locally Bi-directional Mamba" modifies the scan kernel by embedding a **lightweight locally backward scan inside the forward selective scan** and alternating scan directions across layers [2506.15976]. SecMamba does not perform this local bi-directional scan. "Block-Biased Mamba for Long-Range Sequence Processing" changes the selective SSM itself through **block-wise selective dynamics** and a **channel-specific bias** in $\text{B}_2\text{S}_6$ [2505.09022]. SecMamba does not adopt that parameterization. "Mamba-3: Improved Sequence Modeling using State Space Principles" introduces **exponential-trapezoidal discretization**, a **complex-valued state transition**, and an optional **MIMO** SSM core [2603.15569]. Again, SecMamba does not make that class of recurrence-level modification. "TransMamba: Flexibly Switching between Transformer and Mamba" unifies attention and Mamba through **shared parameter matrices** and **TransPoint** scheduling [2503.24067], whereas SecMamba keeps a conventional stage-level separation between Mamba-style blocks and self-attention blocks. "Spectral Informed Mamba for Robust Point Cloud Processing" preserves the Mamba block and changes the **ordering and traversal** of tokens through Laplacian spectral methods [2503.04953]; SecMamba does not redefine token order.

This contrast matters because the name *SecMamba* can otherwise be misread as denoting a general Mamba variant with altered SSM internals. The evidence here suggests a narrower definition: **SecMamba is primarily an architectural wrapper around a MambaVision mixer that adds SeqMoE-based adaptive depth mixing** [2507.17662].

The main limitations reported or implied for SecMamba are correspondingly architectural rather than dynamical. The gating mixes only **two experts at each step**, namely the current expert and the previous aggregated representation. It uses **global mean pooled summaries**, not an explicitly token-wise routing rule. The paper describes **no explicit MoE-specific regularization**, such as load balancing, entropy regularization, routing temperature control, or capacity constraints [2507.17662]. It also identifies broader application-level directions for future work, including **missing views**, **variable breast density**, **limited tumor annotations**, and **multi-task setups** such as joint calcification and mass classification [2507.17662].

Taken together, these characteristics place SecMamba in a specific niche within the Mamba ecosystem: it is a **task-specialized, depth-routed, MambaVision-derived expert block** for high-resolution mammographic sequence modeling, rather than a generic redesign of selective SSM dynamics.

Source: https://www.emergentmind.com/topics/secmamba-block