PanopMamba: SSM-Driven Vision Frameworks

• PanopMamba is a set of advanced SSM-based frameworks that enable unified semantic and instance segmentation along with high-fidelity cross-modal fusion.
• It employs innovative Mamba modules, including MI-SSM and adaptive parameter learning, to achieve efficient, linear-complexity long-range modeling.
• Empirical results show that PanopMamba outperforms attention-based and convolution-only models in nuclei segmentation, pan-sharpening, and 360° image out-painting tasks.

PanopMamba refers to a set of advanced state space model (SSM)-driven deep learning frameworks designed for vision tasks that require panoptic (i.e., unified semantic and instance-wise) reasoning, high-fidelity cross-modal fusion, or spatially coherent generation. Multiple independently developed architectures carry this designation across different application domains, unified by their use of Mamba or Visual Mamba modules to achieve efficient, long-range modeling with linear complexity. The PanopMamba name encompasses: (i) a nuclei panoptic segmentation framework, (ii) a cross-modal pan-sharpening/image enhancement system, and (iii) a text-guided 360° image out-painting method—all utilizing SSMs to surpass the limitations of attention-based or convolution-only counterparts (Wang et al., 17 Dec 2025, He et al., 2024, Gao et al., 2024, Kang et al., 23 Jan 2026).

1. State Space Models and the Mamba Architecture

The core component underlying all PanopMamba variants is the state space model (SSM), formalized as:

$h'(t) = A h(t) + Bx(t),\quad y(t) = C h(t)$

with $h(t)$ as the hidden state, $x(t)$ the input, and parameter matrices ($A,B,C$) dictating information flow and memory. Mamba extends the SSM construction by (i) learning token- and channel-wise parameters ($\overline{A},\overline{B},C$) as functions of the inputs, and (ii) incorporating selective scans for long-range, linear-complexity, contextually adaptive processing.

In vision, such as in Visual Mamba and hierarchical MSVMamba backbones, SSMs are deployed with flattened or patchwise 2D spatial sequences, often with bidirectional or multi-scale scan orderings, enabling information propagation at arbitrary distances with strong efficiency guarantees (He et al., 2024, Wang et al., 17 Dec 2025, Kang et al., 23 Jan 2026).

2. Cross-Modal Fusion for Pan-Sharpening and Super-Resolution

PanopMamba in the remote sensing domain corresponds to the MMMamba framework for pan-sharpening and zero-shot image enhancement (Wang et al., 17 Dec 2025). Given a high-resolution panchromatic (PAN) image $I_p\in\mathbb R^{H\times W\times 1}$ and an upsampled multispectral (MS) image $I_{ms}\in\mathbb R^{H\times W\times C}$, MMMamba employs two Hornet-style gated-convolution encoders, producing features $F_p$ and $F_{ms}$. Fusion is achieved via stacked MMMamba blocks, each structured as follows:

• Projection & DWConv: $$F_{\mu}^{ln} = \text{Linear}(\text{LN}(F_\mu)), F_{\mu}^{silu} = \text{SiLU}(\text{DWConv}(F_{\mu}^{ln}))$$
• Multimodal Interleaved SSM (MI-SSM): Inputs are patch-tokenized and interleaved along four scan directions, subjected to MI_Scan (local window SSMs), and aggregated to yield cross-modal context $S_{mi1}^{out}, S_{mi2}^{out}$.
• Gated Residual Update: $$F_{ms}^{mm} = \text{LN}(S_{mi1}^{out}) \odot \text{SiLU}(F_{ms}^{ln}), F_p^{mm} = \text{LN}(S_{mi2}^{out}) \odot \text{SiLU}(F_p^{ln})$$

Decoding operates via a gated-convolution head, yielding final HRMS output as $I_{hms} = I_{ms} + \Delta I_{ms}$. Notably, MI-SSM-based fusion achieves $O(N)$ compute/memory, contrasting the $O(N^2)$ scaling of attention (Wang et al., 17 Dec 2025).

Training is conducted under $L_1$ loss ($\|\cdot\|_1$) on Wald protocol splits of WorldView-II, GaoFen-2, and WorldView-III. MMMamba achieves the highest PSNR (42.31 dB), SSIM (0.9733), lowest SAM (0.0209), and ERGAS (0.8888) across established benchmarks, outperforming contemporary alternatives such as Pan-Mamba and CFLIHPs.

A zero-shot MS super-resolution extension is realized by omitting the PAN input at test time. The MI-SSM remains operational by inputting duplicated MS tokens, thus leveraging learned in-context fusion representations. This yields superior performance over bicubic, SFINet++, and Pan-Mamba baselines (PSNR=36.49 dB on WV2, SSIM=0.9114).

3. Nuclei Panoptic Segmentation with Multiscale SSM Fusion

In computational pathology, PanopMamba refers to a hybrid encoder-decoder design for nuclei panoptic segmentation (Kang et al., 23 Jan 2026). The encoder is a multiscale VMamba (MSVMamba) stack, featuring patch partitioning and successive MS3 blocks (each Stage combining LayerNorm, DWConv, and MS2D SSM scanning, with SE and linear projections). Downsampling stages create feature maps at $\{1/8,1/16,1/32\}$ spatial resolutions ($\{F_3, F_4, F_5\}$).

A dedicated SSM-based fusion network enhances these features:

• Each fused feature map ($P_l$) results from upsampled and bottlenecked lower-resolution features (FPN style), further refined by SSM modules (applied to flattened patches), LDCNet texture pooling, and efficient channel attention (ECA).

The decoder comprises a pixel-wise and Transformer (Mask2Former-style) head for producing segmentation logits and final class/mask predictions.

To tackle nuclei-specific challenges, PanopMamba introduces alternative Panoptic Quality metrics:

• Image-level PQ ($i$PQ): Averaged per-image PQ, accounting for missing classes.
• Boundary-weighted PQ ($w$PQ): Employs boundary IoU weighted by contour factor $a>1$.
• Frequency-weighted PQ ($fw$PQ): Weights PQ by class instance frequency.

On the MoNuSAC2020 and NuInsSeg benchmarks, PanopMamba attains PQ=73.1%/73.7%, with consistent dominance across all PQ variants relative to Mask2Former, OneFormer, HoVer-Net, and CellViT (e.g., $\sim$+33pp PQ over Mask2Former on MoNuSAC2020). Ablation studies reveal a $\sim$35% PQ drop if SSM-fusion is removed, underlining its centrality.

4. 360-Degree Text-Guided Panorama Out-Painting

The PanopMamba framework also denotes the OPa-Ma/OPaMamba model for generating 360° panoramas from narrow field-of-view (NFoV) images and free-form text (Gao et al., 2024). OPa-Ma incorporates:

• A latent diffusion U-Net backbone (frozen weights)
• Two custom Mamba modules for conditioning:
  • Visual-Textual Consistency Refiner (VCR): Takes CLIP visual/text embeddings, refines them with 1D Mamba SSM, and generates a gated, contextually modulated token stream.
  • Global-Local Mamba Adapter (GMA): Merges global context from cube faces and local NFoV patches using 2D Mamba SSMs to generate multi-scale visual feature maps supplied to the diffusion U-Net.

Generation involves iteratively extracting partially known windows, processing them through VCR and GMA, reconstructing the missing panorama content, and reinserting predictions until the equirectangular image is complete.

On the LAVAL Indoor and Outdoor datasets, OPa-Ma yields substantial performance gains—e.g., FID 9.58 (NFoV+Text) vs. AOG-Net 38.60; qualitative evaluations confirm improved semantic consistency and artifact reduction.

5. Comparative Results and Empirical Validation

PanopMamba models demonstrate consistent superiority across benchmarks and domains, as summarized below:

Application	Key Metric	PanopMamba Value	Best Baseline	Reference
Pan-sharpening	PSNR (WV2 avg)	42.31 dB	42.24 (Pan-Mamba)	(Wang et al., 17 Dec 2025, He et al., 2024)
Nuclei segmentation	PQ (MoNuSAC2020)	73.1%	51.3 (HoVer-Next)	(Kang et al., 23 Jan 2026)
360° out-painting	FID (LAVAL Indoor)	7.60 (NFoV+Text)	9.76 (AOG-Net)	(Gao et al., 2024)

Ablations across all domains confirm the necessity of Mamba-based SSM blocks for both efficient information propagation and robust representation learning, with non-SSM architectures suffering notable accuracy drops.

6. Design Considerations, Limitations, and Varieties

PanopMamba architectures universally seek to (i) leverage scalable, adaptive state-space operations, (ii) maximize cross-modal or multiscale feature transfer, and (iii) support domain-specific forms of conditioning (e.g., MI-SSM fusion, VCR+GMA adapters).

Specific limitations include:

• Fixed scan directions in MI-SSM (MMMamba) preclude data-adaptive fusion; learning scan patterns is an open research question.
• In 360° panorama generation, only VCR and GMA are trained; the frozen U-Net backbone may restrict further gains.
• PQ variants in nuclei segmentation are tailored for evaluation bias mitigation, but generalization across tasks may require additional calibration.

A plausible implication is that future PanopMamba variants may generalize to arbitrary-scale super-resolution, other multimodal fusion domains, or more expressive panoptic quality assessment pipelines.

7. Implementation Details and Reproducibility

PanopMamba implementations typically rely on PyTorch, with domain-specific libraries (e.g., MM-Segmentation for nuclei analysis). Backbones are pretrained (e.g., MSVMamba-Tiny on ImageNet-1K), and training leverages AdamW, cosine learning rate decay, and class-specific augmentations (e.g., color jitter, stain normalization for pathology). Annotation follows task-appropriate standards, e.g., COCO panoptic format for segmentation.

Source code for the nuclei PanopMamba is available at https://github.com/mkang315/PanopMamba (Kang et al., 23 Jan 2026). The MMMamba framework for pan-sharpening and zero-shot super-resolution is detailed in (Wang et al., 17 Dec 2025), while OPa-Ma out-painting and Pan-Mamba pan-sharpening resources can be found at (Gao et al., 2024, He et al., 2024).