Modality-and-Slice Memory Attention (MSMA)

• Modality-and-Slice Memory Attention (MSMA) is a dual-memory mechanism that leverages cross-modal and inter-slice contexts to enhance feature representation in brain tumor segmentation.
• It dynamically constructs two external memories from prior predictions—one for modality context and one for slice continuity—to guide segmentation.
• Empirical results show that MSMA improves segmentation accuracy with moderate latency overhead, making it a practical solution for multi-modal MRI analysis.

Searching arXiv for the cited MSM-Seg paper and the SAM2 work it references. {"query":"(Luo et al., 12 Oct 2025) MSM-Seg A Modality-and-Slice Memory Framework with Category-Agnostic Prompting for Multi-Modal Brain Tumor Segmentation", "max_results": 5} Modality-and-Slice Memory Attention (MSMA) is a dual-memory cross-attention module introduced within the MSM-Seg framework for multi-modal brain tumor segmentation. Its purpose is to exploit two forms of complementary context that are central in multi-modal MRI yet often only partially modeled: cross-modal complementarities across T1, T1c, T2, and FLAIR, and inter-slice continuity along the scan axis. In MSM-Seg, MSMA is the mechanism that transforms past, context-rich predictions into two dynamic external memories—one over modalities and one over slices—and uses them to enhance the current slice-modality embedding through dedicated cross-attention branches (Luo et al., 12 Oct 2025).

1. Motivation and problem setting

Multi-modal brain tumor segmentation requires accurate identification of distinct internal anatomical subregions, and the underlying imaging context is distributed both across modalities and across neighboring slices. The MSM-Seg formulation identifies two limitations in prior work: many 3D CNN/Transformer variants process full volumes with high computational cost and often ignore sequential slice dependencies, while recent prompt-based paradigms rely on category-specific prompts and typically treat modalities independently or via naive concatenation (Luo et al., 12 Oct 2025).

MSMA is designed to address these limitations by explicitly modeling both cross-modal and inter-slice relationships. The central idea is not merely to attend within the current feature map, but to query two external memories that are dynamically constructed from the model’s own predictions. One memory stores modality-specific context within the current slice; the other stores fused slice-level context from previous slices. This yields a context-aware representation for the current slice-modality pair without introducing fixed learned memory slots.

Within the broader MSM-Seg framework, MSMA is paired with efficient category-agnostic prompting. The paper states that this design avoids reliance on category-specific annotations as prompts and supports both a single bounding-box prompt for the whole tumor and a fully automatic mode (Luo et al., 12 Oct 2025). A plausible implication is that MSMA should be understood not as an isolated attention block, but as the context-integration component that makes category-agnostic prompting operational in a multi-modal setting.

2. Formal setup and memory construction

The MSM-Seg formulation indexes slices by $t \in \{1, \ldots, T\}$ and modalities by $m \in \{1, \ldots, M\}$, where the standard four MRI sequences are T1, T1c, T2, and FLAIR. At step $(t,m)$, the model receives a single 2D slice $X_{t,m}$ and maintains a latent state $S_{t,m}$ that summarizes both cross-modal context within the current slice and inter-slice context from past slices:

$$S_{t,m} = R(X_{t,m}, \theta_{t,m}, S_{t,\prec m}, S_{\prec t}), \qquad \hat{Y}_{t,m} = P(S_{t,m}).$$

Here, $S_{t,\prec m}$ aggregates past modalities at the same slice, and $S_{\prec t}$ aggregates past slices. The input slice is encoded by a pretrained image encoder $E$—specifically Hiera-S initialized from SAM2—to produce an image embedding $F_{t,m} \in \mathbb{R}^{C \times H \times W}$ (Luo et al., 12 Oct 2025).

The two external memory banks are dynamic and data-dependent. For the modality memory, each modality-specific prediction $m \in \{1, \ldots, M\}$0 is encoded as

$m \in \{1, \ldots, M\}$1

and the modality memory for slice $m \in \{1, \ldots, M\}$2 is

$m \in \{1, \ldots, M\}$3

with capacity $m \in \{1, \ldots, M\}$4. For the slice memory, after modalities are fused into a slice-level prediction $m \in \{1, \ldots, M\}$5, the fused prediction is encoded as

$m \in \{1, \ldots, M\}$6

and the slice memory is

$m \in \{1, \ldots, M\}$7

with capacity $m \in \{1, \ldots, M\}$8 (Luo et al., 12 Oct 2025).

A defining property of these memories is that they are populated online from current and past predictions during training and inference. The paper emphasizes that they are not fixed learned parameters, but encodings of predictions via $m \in \{1, \ldots, M\}$9 and $(t,m)$0. Default values used in experiments are $(t,m)$1 for modality memory and $(t,m)$2 for slice memory (Luo et al., 12 Oct 2025).

3. Core architecture and exact formulation

MSMA consumes the current image embedding $(t,m)$3 together with the slice memory $(t,m)$4 and the modality memory $(t,m)$5, and produces a memory-enhanced embedding $(t,m)$6. The architecture is defined by Eqs. (3)–(5) in the MSM-Seg paper (Luo et al., 12 Oct 2025).

The first operation is an even split of the encoder feature along the channel dimension:

$(t,m)$7

Given $(t,m)$8, this yields

$(t,m)$9

The split induces two disjoint query paths. One path is dedicated to slice memory, and the other to modality memory.

Each branch then applies a $X_{t,m}$0 projection $X_{t,m}$1 followed by self-attention refinement:

$X_{t,m}$2

In this notation, $X_{t,m}$3 denotes self-attention. The role of $X_{t,m}$4 is up-projection/alignment, and the self-attention serves to refine query features before memory retrieval (Luo et al., 12 Oct 2025).

The central operation is dual cross-attention into the two external memories:

$X_{t,m}$5

Here, $X_{t,m}$6 denotes cross-attention. The slice branch uses the slice memory as both keys and values, and the modality branch uses the modality memory as both keys and values. The outputs of the two branches are fused by addition to yield the final memory-enhanced embedding $X_{t,m}$7 (Luo et al., 12 Oct 2025).

This formulation establishes the key novelty of MSMA: two distinct, external, dynamically updated memories are queried by two dedicated query streams derived from disjoint channel partitions of the current feature. The paper does not specify additional positional or relative index encodings inside MSMA, nor does it define extra slice-index encodings or cross-slice neighborhood terms beyond those implicit in the Hiera-S/SAM2 backbone. The number of heads $X_{t,m}$8 and embedding dimension $X_{t,m}$9 are also not explicitly specified; attention configurations are inherited from the backbone (Luo et al., 12 Oct 2025).

4. Functional role inside MSM-Seg

MSMA is the first stage after the image encoder in each $S_{t,m}$0 step of MSM-Seg. The step-wise data flow is specified in Algorithm 1 and summarized in the paper as follows: first extract features with $S_{t,m}$1, then retrieve the two memories, compute the memory-enhanced embedding with MSMA, pass that embedding to the multi-scale category-agnostic prompt encoder (MCP-Encoder), decode with the modality-adaptive fusion decoder (MF-Decoder), and finally update both memories with the newly generated predictions (Luo et al., 12 Oct 2025).

In this pipeline, MSMA provides the context-aware feature representation that informs downstream prompting and decoding. The MCP-Encoder receives the set of encoder features $S_{t,m}$2 together with $S_{t,m}$3 and constructs a multi-scale embedding group. Its layer-wise formulation is

$S_{t,m}$4

and its region-guidance head is

$S_{t,m}$5

In this expression, $S_{t,m}$6 is two $S_{t,m}$7 convolutions followed by one $S_{t,m}$8 convolution, $S_{t,m}$9 is sigmoid, and $$S_{t,m} = R(X_{t,m}, \theta_{t,m}, S_{t,\prec m}, S_{\prec t}), \qquad \hat{Y}_{t,m} = P(S_{t,m}).$$0 serves as the whole-tumor category-agnostic region guidance (Luo et al., 12 Oct 2025). The paper states that $$S_{t,m} = R(X_{t,m}, \theta_{t,m}, S_{t,\prec m}, S_{\prec t}), \qquad \hat{Y}_{t,m} = P(S_{t,m}).$$1 contributes to MCP-Encoder’s embedding group, so the memory-enhanced representation produced by MSMA informs prompt generation at multiple scales.

The MF-Decoder then fuses the memory-enhanced embedding and the prompt guidance. It combines prompt and image embeddings by element-wise addition,

$$S_{t,m} = R(X_{t,m}, \theta_{t,m}, S_{t,\prec m}, S_{\prec t}), \qquad \hat{Y}_{t,m} = P(S_{t,m}).$$2

and generates the modality-specific prediction via

$$S_{t,m} = R(X_{t,m}, \theta_{t,m}, S_{t,\prec m}, S_{\prec t}), \qquad \hat{Y}_{t,m} = P(S_{t,m}).$$3

The modality-specific outputs are combined with adaptive weights to obtain the fused slice-level mask:

$$S_{t,m} = R(X_{t,m}, \theta_{t,m}, S_{t,\prec m}, S_{\prec t}), \qquad \hat{Y}_{t,m} = P(S_{t,m}).$$4

Each $$S_{t,m} = R(X_{t,m}, \theta_{t,m}, S_{t,\prec m}, S_{\prec t}), \qquad \hat{Y}_{t,m} = P(S_{t,m}).$$5 updates the modality memory, and the fused output $$S_{t,m} = R(X_{t,m}, \theta_{t,m}, S_{t,\prec m}, S_{\prec t}), \qquad \hat{Y}_{t,m} = P(S_{t,m}).$$6 updates the slice memory, thereby closing the dual-memory loop (Luo et al., 12 Oct 2025). This suggests that MSMA functions as the recurrent context interface of MSM-Seg: it converts accumulated prediction history into information that can be reused at every subsequent slice-modality step.

5. Optimization, efficiency, and empirical behavior

The training objective for MSM-Seg jointly optimizes prompt guidance, modality-specific predictions, and fused masks:

$$S_{t,m} = R(X_{t,m}, \theta_{t,m}, S_{t,\prec m}, S_{\prec t}), \qquad \hat{Y}_{t,m} = P(S_{t,m}).$$7

The paper specifies that $$S_{t,m} = R(X_{t,m}, \theta_{t,m}, S_{t,\prec m}, S_{\prec t}), \qquad \hat{Y}_{t,m} = P(S_{t,m}).$$8 is binary cross-entropy, while $$S_{t,m} = R(X_{t,m}, \theta_{t,m}, S_{t,\prec m}, S_{\prec t}), \qquad \hat{Y}_{t,m} = P(S_{t,m}).$$9 and $S_{t,\prec m}$0 are Dice + focal losses. Optimization uses AdamW with initial learning rate $S_{t,\prec m}$1, weight decay $S_{t,\prec m}$2, exponential LR decay factor $S_{t,\prec m}$3, batch size $S_{t,\prec m}$4, and $S_{t,\prec m}$5 epochs. The encoder is Hiera-S initialized with SAM2 weights, and the memory banks are updated online as described in Algorithm 1 (Luo et al., 12 Oct 2025).

Although the paper does not provide a closed-form time or space complexity for MSMA, it states that the module uses two cross-attentions per step and that cost scales with the number of query tokens and total memory length, with $S_{t,\prec m}$6 and $S_{t,\prec m}$7 (Luo et al., 12 Oct 2025). The reported ablations show moderate empirical overhead as memory size increases.

For modality memory, increasing $S_{t,\prec m}$8 from $S_{t,\prec m}$9 in automatic mode improves Avg Dice on BraTS-METS from $S_{\prec t}$0 and on BraTS-AGPT from $S_{\prec t}$1, while per-slice latency rises from $S_{\prec t}$2 s to $S_{\prec t}$3 s. For slice memory, increasing $S_{\prec t}$4 from $S_{\prec t}$5 improves Avg Dice from $S_{\prec t}$6 on BraTS-METS and from $S_{\prec t}$7 on BraTS-AGPT, with latency increasing from $S_{\prec t}$8 s to $S_{\prec t}$9 s (Luo et al., 12 Oct 2025).

The paper further notes that strong accuracy gains are obtained up to the default settings $E$0 and $E$1, after which performance saturates while latency increases more noticeably. It remarks that this is consistent with the clinical prior that tumors span a limited number of slices and exhibit modality consistency (Luo et al., 12 Oct 2025). Since no explicit failure cases are reported, this observation should be read as an empirical characterization rather than a formal limitation analysis.

6. Ablation evidence, comparative positioning, and interpretation

The direct contribution of MSMA is isolated in the ablation study reported in Table 2. Starting from the SAM2 baseline, adding only MSMA increases BraTS-METS Avg Dice from $E$2 to $E$3 and improves Avg HD95 from $E$4 to $E$5 mm. On BraTS-AGPT, the same change increases Avg Dice from $E$6 to $E$7 and improves Avg HD95 from $E$8 to $E$9 mm (Luo et al., 12 Oct 2025). The paper states that these gains are consistent across ET/TC/WT regions, supporting the claim that explicitly attending to modality and slice memories improves segmentation.

Additional studies indicate that both forms of memory are beneficial. The modality and slice capacity ablations show monotonic improvements as $F_{t,m} \in \mathbb{R}^{C \times H \times W}$0 and $F_{t,m} \in \mathbb{R}^{C \times H \times W}$1 increase, and the modality-order study reports that multiple permutations of $F_{t,m} \in \mathbb{R}^{C \times H \times W}$2 yield statistically indistinguishable performance with $F_{t,m} \in \mathbb{R}^{C \times H \times W}$3 (Luo et al., 12 Oct 2025). This suggests that the cross-modal attention mechanism is robust to input scheduling rather than being tied to a specific modality order.

The paper positions MSMA against several adjacent designs. Relative to standard self-attention or non-local blocks, MSMA augments intra-feature dependency modeling with two external dynamically updated memories and performs cross-attention into each. Relative to classic cross-modal transformers, which often concatenate or exchange features across modalities in a single pass, MSMA structures cross-modal context as a selective, capacity-limited memory formed from recent predictions. Relative to SAM2 memory attention, which maintains a temporal memory for frames, MSMA introduces a second parallel memory bank specific to modalities within the same slice and fuses the two memory attentions by summation (Luo et al., 12 Oct 2025).

These comparisons clarify the conceptual status of MSMA. It is neither a generic self-attention block nor a conventional multi-modal fusion transformer. Its defining principle is prediction-derived dual memory: cross-modal and inter-slice context are externalized, updated online, and retrieved by separate query streams. In the MSM-Seg framework, that principle underwrites both the integration of complementary MRI information and the use of category-agnostic prompting for metastases and glioma tumor segmentation (Luo et al., 12 Oct 2025).