MemorySAM for Multi-Modal Fusion

• The paper demonstrates that MemorySAM integrates multi-modal inputs via sequential memory attention, enabling effective dense segmentation across diverse sensors.
• The framework introduces a semantic prototype memory module that blends instance-level details with class-level semantics to enhance prediction accuracy.
• Empirical results report significant mIoU improvements on challenging benchmarks using modalities like RGB, depth, event, LiDAR, and polarization cues.

MemorySAM is a framework for leveraging the Segment Anything Model 2 (SAM2) to perform multi-modal fusion for dense prediction tasks, notably multi-modal semantic segmentation and multi-modal video salient object detection. By repurposing SAM2’s streaming memory originally designed for video, MemorySAM enables the aggregation of modality-agnostic scene information while also incorporating mechanisms for infusing semantic class knowledge. This approach has been shown to yield large quantitative improvements on challenging benchmarks involving diverse modality combinations, such as RGB, depth, event, LiDAR, and polarization cues (Liao et al., 9 Mar 2025, Liu et al., 12 May 2026).

1. Core Principles of MemorySAM for Multi-Modal Fusion

MemorySAM reconceptualizes multi-modal fusion by treating each modality as a “frame” in a virtual sequence, applying SAM2’s memory-augmented attention to integrate modality-agnostic information across the entire input set. This temporalization of modalities facilitates the transfer of SAM2’s proven video reasoning capabilities to the multi-modal domain, distinguishing MemorySAM from standard feature concatenation or late fusion methods.

The pivotal innovations are:

• Sequential application of memory attention across modalities.
• Storage and refinement of a shared scene representation incrementally, as each modality is processed.
• Optional semantic prototype memory—injecting class-level prototypes learned during training to bridge the gap between instance-based and semantic segmentation.

These innovations result in a system that robustly handles heterogeneity in input sources while maintaining strong semantic discrimination (Liao et al., 9 Mar 2025).

2. Architecture and Memory Mechanisms

Multi-Modal Memory Flow

Given $M$ modalities $\{x^1, x^2, \ldots, x^M\}$ (e.g., RGB, depth, event, LiDAR), MemorySAM arranges them as an ordered sequence: $\overline{x} = [x^1, x^2, \ldots, x^M]$. For each index $i$:

• $x^i$ is independently encoded by a LoRA-fine-tuned SAM2 image encoder, yielding multi-scale features ($F_e^i$) and higher-resolution decoder features ($F_\mathrm{high1}^i$, $F_\mathrm{high2}^i$).
• For the first modality, the memory attention is bypassed: $F_c^1 = F_e^1$.
• For subsequent modalities ($i>1$), encoded features from all previous “frames” are concatenated and attended to via SAM2’s multi-head memory attention:

$\{x^1, x^2, \ldots, x^M\}$0

Here, $\{x^1, x^2, \ldots, x^M\}$1 and $\{x^1, x^2, \ldots, x^M\}$2 are, respectively, the feature and positional memory outputs from previous modalities.

• The output $\{x^1, x^2, \ldots, x^M\}$3 is fed to the mask decoder, generating a dense mask $\{x^1, x^2, \ldots, x^M\}$4.

Memory Update and Key Equations

After producing $\{x^1, x^2, \ldots, x^M\}$5, a memory decoder encodes it to form the next memory key/value pair, which is appended to the running memory bank:

$\{x^1, x^2, \ldots, x^M\}$6

This sequential memory mechanism ensures that every modality can reference and update a shared representation, resulting in a fused, modality-invariant depiction of the scene (Liao et al., 9 Mar 2025).

3. Integration of Semantic Prototype Memory

Standard SAM2 provides strong instance segmentation capabilities but does not learn explicit semantic class representations. MemorySAM introduces a training-only Semantic Prototype Memory Module (SPMM) to inject broader category knowledge:

• For each class $\{x^1, x^2, \ldots, x^M\}$7, a per-class “current” prototype $\{x^1, x^2, \ldots, x^M\}$8 is computed by spatially averaging projected features over all modalities masked to class $\{x^1, x^2, \ldots, x^M\}$9.
• A “global” prototype $\overline{x} = [x^1, x^2, \ldots, x^M]$0 is maintained using momentum:

$\overline{x} = [x^1, x^2, \ldots, x^M]$1

• Each iteration applies a prototypical adaptation loss to align current and global prototypes:

$\overline{x} = [x^1, x^2, \ldots, x^M]$2

• The training objective is:

$\overline{x} = [x^1, x^2, \ldots, x^M]$3

This module is deactivated at inference, so MemorySAM’s test-time cost is unchanged. The SPMM structure aligns local and global semantics, facilitating the transition from instance-centric to class-centric dense prediction (Liao et al., 9 Mar 2025).

4. Specialized Multi-Modal Fusion Strategies in Downstream Tasks

Beyond the generic architecture, MemorySAM underpins more specialized applications such as M$\overline{x} = [x^1, x^2, \ldots, x^M]$4-SAM (Liu et al., 12 May 2026) for RGB-D video salient object detection, incorporating further multi-modal and temporal modeling innovations:

• Modality-Aware MoE-LoRA: Extends classic LoRA by embedding a convolutional Mixture-of-Experts module in SAM2’s encoder, with discrete expert groups for RGB, depth, and shared fusion. The Modality Dispatcher $\overline{x} = [x^1, x^2, \ldots, x^M]$5 selectively activates expert groups based on input modality:

$\overline{x} = [x^1, x^2, \ldots, x^M]$6

• Gated Multi-Level Feature Fusion (Gated-MLF): Aggregates hierarchical encoder features, applies spatial and channel-wise attention, calculates a learned gate $\overline{x} = [x^1, x^2, \ldots, x^M]$7, and fuses with decoder outputs for improved spatial-semantic balance.
• Prompt-Free Pseudo-Guided Initialization: Eliminates the need for manual prompts by generating a pseudo-prior from an early-coarse mask, seeding the temporal memory for zero-shot inference.
• Temporal Fusion and Memory Update: Standardizes memory update via key/value projections and attention, integrating the outputs of multi-modal feature fusion across video frames.

Step-wise, the process enables dense, accurate, and prompt-free detection and segmentation on RGB-D video with strong empirical gains (Liu et al., 12 May 2026).

5. Comparative Performance and Quantitative Results

MemorySAM and its derivatives demonstrate superior empirical performance in a range of MMSS and VSOD settings:

• On the DELIVER dataset (RGB + Depth + Event + LiDAR), MemorySAM achieves 65.38 mIoU, over 6% above prior state-of-the-art methods.
• On MCubeS (RGB + AoLP + DoLP + NIR), MemorySAM attains 52.88 mIoU, a 16.7% gain over previous best baselines (Liao et al., 9 Mar 2025).
• Ablation studies confirm that sequential memory improves mIoU by nearly 3% over baseline; inclusion of SPMM further enhances performance.
• Qualitative analyses (e.g., t-SNE feature distributions) show that memory attention results in tighter intra-class feature clusters and clearer inter-class separation.

Experiments on RGB-D video VSOD tasks with M$\overline{x} = [x^1, x^2, \ldots, x^M]$8-SAM also report state-of-the-art results across multiple public datasets, illustrating optimal retention of spatial details and semantic context (Liu et al., 12 May 2026).

6. Training Paradigms, Implementation, and Broader Implications

MemorySAM employs a LoRA-fine-tuned SAM2 encoder with memory encoder/attention and mask decoder. Only LoRA modules are trained in the backbone; the rest of SAM2 remains frozen. Data augmentation strategies mirror standard practices in the field. Inference efficiency is retained by utilizing memory mechanisms only, with SPMM engaged solely during training (Liao et al., 9 Mar 2025).

The approach successfully translates cross-frame correlation methods from temporal video modeling to multi-modal spatial fusion, providing a unifying memory-based substrate for diverse sensor modalities. The framework’s prompt-free, modular structure enables scalable extension to further modalities or additional semantic constraints without sacrificing inference tractability—potentially generalizing to broader categories of multi-modal dense prediction problems.

7. Context, Limitations, and Future Directions

MemorySAM’s formulation, especially its “modality as sequence” abstraction and integration of class prototypes, represents a robust step toward modality-agnostic scene understanding. Limitations include the necessity of sequential computation (with complexity linear in the modality count) and the absence of explicit modeling for interactions between modalities beyond memory attention. In scenarios where modalities are heavily redundant or adversarial, alternative cross-modal interaction mechanisms may be required. Ongoing research explores memory compression, dynamic modality selection, and continued adaptation of large vision foundation models for multi-modal, prompt-free dense prediction (Liao et al., 9 Mar 2025, Liu et al., 12 May 2026).