--- title: Hierarchical Gated Mamba Fusion url: https://www.emergentmind.com/topics/hierarchical-gated-mamba-fusion-gm-fusion type: topic --- # Hierarchical Gated Mamba Fusion Hierarchical Gated Mamba Fusion (GM-Fusion) denotes a family of multimodal fusion architectures that couple hierarchical feature processing with Mamba-style selective state-space models and learned gates over modalities, scales, branches, or experts. In the cited literature, the term appears explicitly in GMF-Drive and as a direct architectural extension of GraspMamba; related systems realize the same pattern through entropy-weighted branch fusion, reliability-aware Bird’s Eye View (BEV) gating, event-density-conditioned projection gates, expert routing, or cross-modal state-space scanning [2508.06113][2409.14403]. ## 1. Definition and canonical formulation A canonical GM-Fusion block takes multi-scale visual features \(\{\mathbf{X}_l\}_{l=1}^L\) and a conditioning representation \(\mathbf{T}\), aligns them into a common latent space, computes an explicit gate, applies a Mamba-based fusion operator, and then aggregates the fused outputs hierarchically. In the generalized formulation derived from GraspMamba, the inputs are visual features \(\mathbf{X}_l \in \mathbb{R}^{B \times C_l \times H_l \times W_l}\), a text embedding \(\mathbf{T} \in \mathbb{R}^{B \times C_T}\), a Mamba-based fusion block \(\mathcal{M}_l\), and a fusion gate \(g_l\). The per-scale computation is written as \[ \begin{aligned} \mathbf{X}_l' &= \mathrm{Conv}_{1\times1}^{\mathbf{X}_l}(\mathbf{X}_l), \\ \mathbf{T}_{\mathrm{exp},l} &= \mathrm{Broadcast}(\mathbf{T}), \\ \mathbf{T}_l' &= \mathrm{Conv}_{1\times1}^{\mathbf{T}}(\mathbf{T}_{\mathrm{exp},l}), \\ g_l &= \sigma\!\left(\mathrm{Conv}_{1\times1}^{g}\big([\mathbf{X}_l',\mathbf{T}_l']\big)\right), \\ \mathbf{Z}_l &= g_l \cdot \mathbf{T}_l' + (1-g_l)\cdot \mathbf{X}_l'. \end{aligned} \] After flattening spatial dimensions, \[ \mathbf{Z}_l^{\mathrm{seq}} \in \mathbb{R}^{B \times (H_lW_l) \times D}, \qquad \mathbf{F}_l^{\mathrm{seq}} = \mathcal{M}_l(\mathbf{Z}_l^{\mathrm{seq}}), \] and the result is reshaped back to a feature map. Hierarchical aggregation is then defined as \[ F_l^{\mathrm{hier}} = \begin{cases} F_L, & l=L, \\ F_l + \alpha_l \cdot U_{l+1}(F_{l+1}^{\mathrm{hier}}), & 1 \le l < L, \end{cases} \] where \(U_{l+1}\) is bilinear upsampling plus a \(3\times 3\) convolution and \(\alpha_l\) is a learned scalar gate per level [2409.14403]. This formulation separates three distinct design axes. First, it is hierarchical because it operates over a pyramid \(\{1,\dots,L\}\) rather than a single latent level. Second, it is gated because modality mixing and top-down aggregation are modulated by learned coefficients such as \(g_l\) and \(\alpha_l\). Third, it is Mamba-based because the core token mixer is a selective state-space model rather than a quadratic self-attention layer. The same decomposition recurs across later systems, even when the conditioning modality is not text and the output head is not task-specific. ## 2. State-space foundations and Mamba operators GM-Fusion inherits its computational core from structured state-space models. The standard continuous-time formulation used across the surveyed systems is \[ \mathbf{h}'(t)=\mathbf{A}\mathbf{h}(t)+\mathbf{B}\mathbf{x}(t), \qquad \mathbf{y}(t)=\mathbf{C}\mathbf{h}(t), \] with the corresponding discretization \[ \overline{\mathbf{A}}=\exp(\Delta \mathbf{A}), \qquad \overline{\mathbf{B}}=(\Delta \mathbf{A})^{-1}\big(\exp(\Delta \mathbf{A})-\mathbf{I}\big)\cdot \Delta \mathbf{B}, \] and recurrence \[ \mathbf{h}_t=\overline{\mathbf{A}}\mathbf{h}_{t-1}+\overline{\mathbf{B}}\mathbf{x}_t, \qquad \mathbf{y}_t=\mathbf{C}\mathbf{h}_t. \] In Mamba, the effective state update becomes selective because \(\Delta\), \(\mathbf{B}\), and \(\mathbf{C}\) are input-dependent; the practical result is linear-complexity sequence modeling with long-range context [2409.15936]. Vision-oriented GM-Fusion variants differ chiefly in how 2D or multi-scale structure is serialized before the selective scan. FusionMamba uses four-directional image flattening and reconstructs the output by summing directional branches, with an explicit multiplicative gate \(\mathbf{Y}\cdot \mathrm{SiLU}(\mathbf{Z})\) in both the single-input Mamba block and the dual-input FusionMamba block [2404.07932]. HiddenObject uses SS2D in Vision Mamba encoders and decoders, rearranging feature maps into sequences under four scan directions and then merging them back to 2D feature maps [2508.21135]. GMF-Drive introduces BEV-specific sequencing, including raster and zigzag scans, and combines them with direction-aware state transition matrices and distance decay in a BEV-SSM [2508.06113]. MambaFusion serializes LiDAR voxels along a Hilbert space-filling curve and applies temporal Mamba over camera BEV embeddings \(B_t\) to aggregate spatiotemporal context in linear time [2602.08126]. WaveMamba applies VMamba-style SS2D in the low-frequency branch after wavelet decomposition, while SMC-Mamba extends scanning to spatial-channel, cross-modal, and frequency-rotational domains [2507.18173][2512.20921]. A central consequence is that “Mamba fusion” does not designate a single operator. In some systems Mamba is only the backbone, with fusion remaining convolutional; in others, the fusion path itself becomes state-space-driven. The distinction is architecturally important and recurs in the literature. ## 3. Gating mechanisms The “gated” component of GM-Fusion is realized in multiple, non-equivalent ways. In the generalized GraspMamba extension, gating is explicit and scalar or spatial: \[ g_l = \sigma\!\left(f_g(\mathbf{X}_l,\mathbf{T})\right)\in[0,1], \] with modality blending \[ \Phi_l^{\mathrm{gated}}=\mathrm{Conv}_{3\times 3}\!\left( g_l \cdot \mathrm{Conv}_{1\times1}^{\mathbf{T}}(\mathbf{T}_{\mathrm{exp}}) + (1-g_l)\cdot \mathrm{Conv}_{1\times1}^{\mathbf{X}}(\mathbf{X}_l) \right), \] and hierarchical gating in the top-down path \[ F_l^{\mathrm{gated}}=\Phi_l^{\mathrm{gated}}+\alpha_l \cdot U_{l+1}(F_{l+1}^{\mathrm{gated}}). \] The original GraspMamba does not include such explicit coefficients: its fusion is deterministic concatenation, \(1\times1\) projection, \(3\times3\) convolution, and recursive addition, and Mamba is used only in the visual backbone [2409.14403]. ME-Mamba uses several implicit gating mechanisms. Within each Attention-based Mamba Layer, every scan branch computes \(Y=\operatorname{SSM}(\cdot)\), \(Z=\operatorname{SiLU}(\operatorname{Linear}(\cdot))\), and then applies the Mamba-style gate \(Z\odot Y\). At cross-modal level, Optimal Transport matrices \(M_{p2g}\) and \(M_{g2p}\) act as token-level gates by selecting the most similar partner in the other modality, while the global MMD loss constrains the distributional space in which fusion is allowed [2509.16900]. SlideMamba makes the gate explicit and confidence-driven. If \(\hat{\mathbf{y}}_{\mathrm{SG}}\) and \(\hat{\mathbf{y}}_{\mathrm{Mamba}}\) are branch probabilities, normalized entropy is \[ H(\hat{\mathbf{y}})=-\frac{1}{\log(C)}\sum_{c=1}^C \hat{y}_c \log(\hat{y}_c), \] confidence is \(w=1-H\), and the fusion coefficient is \[ \alpha=\frac{w_{\mathrm{Mamba}}}{w_{\mathrm{SG}}+w_{\mathrm{Mamba}}}. \] The fused representation is then \[ \mathbf{X}^{l+1}_{\mathrm{SlideMamba}} =(1-\alpha)\mathbf{X}^{l+1}_{\mathrm{SG}}+\alpha \mathbf{X}^{l+1}_{\mathrm{Mamba}}. \] This is a scalar gate per block, derived from predictive entropy rather than from a learned cross-attention matrix [2509.21239]. MambaTrack couples temporal gating and cross-modal gating. Event density \[ p_t=\frac{\mathrm{count}(V_t)}{H\times W} \] is projected into \[ \beta=\sigma(W_a p_t), \] and the event-branch state transition becomes \[ A_t=\alpha \cdot \beta \cdot A_{\mathrm{base}} + (1-\alpha)\cdot A_{t-1}, \qquad A_{\mathrm{final}}=A_t+A. \] At the fusion interface, the Gated Projection Fusion module computes \[ G=\mathrm{Sigmoid}\!\left(W_g[p(t);\|F_{\mathrm{RGB}}\|_2]\right), \] and fuses RGB into event space as \[ F_{\mathrm{fuse}\rightarrow E}=F_{\mathrm{Event}}+G\odot A_F. \] The same design is used symmetrically for event-to-RGB fusion [2604.13426]. MambaFusion adopts a spatially varying, reliability-aware gate. For each BEV cell \((x,y)\), the descriptor \[ \mathbf{g}(x,y)=[\rho_L,\tau_C,o,\phi,d] \] is passed to a small MLP with attended camera and LiDAR BEV tokens: \[ g(x,y)=\sigma\Big(\Phi_{\mathrm{gate}}([\mathbf{g}(x,y),Q_C^{\mathrm{att}}(x,y),Q_L^{\mathrm{att}}(x,y)])\Big). \] Per-cell uncertainties \(\sigma_C^2(x,y)\) and \(\sigma_L^2(x,y)\) then define inverse-variance fusion \[ Q_{\mathrm{fused}}(x,y)= \frac{g\,Q_C^{\mathrm{att}}/\sigma_C^2 + (1-g)\,Q_L^{\mathrm{att}}/\sigma_L^2} {g/\sigma_C^2 + (1-g)/\sigma_L^2 + \epsilon}. \] This combines gating and uncertainty calibration in a single fusion rule [2602.08126]. SMC-Mamba generalizes gating to expert routing. After global average and max pooling over concatenated multimodal features, the expert weights are \[ W_{\mathrm{exp}}=\mathrm{Softmax}\big(\mathrm{TopK}(F_g\cdot W_g + \epsilon)\big), \qquad \epsilon=\mathcal{N}(0,1)\cdot \mathrm{Softplus}(F_g\cdot W_{\mathrm{noise}}), \] and the fused output is \[ F_{mf}=\sum_{i=1}^{N} W_{\mathrm{exp}}^i \cdot \hat{F}_i. \] Inside each expert, cross-modal Mamba outputs are further gated by the opposite modality: \[ F_{mf}^N = F_{cm1}\odot \mathrm{SiLU}(F_{ln2}) + F_{cm2}\odot \mathrm{SiLU}(F_{ln1}). \] The result is a hierarchical gating stack: local gating, global gating, expert gating, and cross-modal gating [2512.20921]. ## 4. Hierarchical structure across scales, branches, and domains The hierarchical aspect of GM-Fusion is not restricted to a feature pyramid; it also appears as cross-scale recursion, wavelet sub-band decomposition, encoder-decoder skip fusion, multi-expert routing, and object-level refinement. GraspMamba is the most direct coarse-to-fine example. Its four-stage CNN–MambaVision backbone yields \(\mathbf{X}_1,\mathbf{X}_2,\mathbf{X}_3,\mathbf{X}_4\), and multimodal fusion is performed at every level. Local fusion \(\Phi_l(\mathbf{X}_l,\mathbf{T})\) is combined with top-down context using \[ F_l= \begin{cases} \Phi_L(\mathbf{X}_L,\mathbf{T}), & l=L, \\ \Phi_l(\mathbf{X}_l,\mathbf{T}) + U_{l+1}(F_{l+1}), & 1\le l