ProMSC-MIS: Prompt-Based Multimodal Segmentation

• The paper introduces a prompt-based framework for multi-spectral image segmentation that transmits semantic content rather than exact image reconstruction.
• It leverages contrastive pre-training, prompt learning, and cross-attention fusion to enhance complementary modality features while reducing channel bandwidth by up to 70%.
• The system optimizes performance for real-world applications like autonomous driving and nighttime surveillance by cutting storage, computation, and bandwidth demands.

Searching arXiv for the cited ProMSC-MIS paper and closely related multimodal semantic communication work. ProMSC-MIS is a prompt-based multimodal semantic communication framework for multi-spectral image segmentation that is designed for task-oriented transmission of spatially aligned RGB and thermal imagery over band-limited channels. Its stated purpose is not faithful image reconstruction in the conventional bit-level sense, but the efficient delivery of semantic content that is critical for downstream segmentation. The framework combines prompt learning, contrastive pre-training, cross-attention-based fusion, squeeze-and-excitation enhancement, and a learnable bit generator to exploit complementary information across modalities while controlling bandwidth, storage, and computation. Reported results indicate that, at the same segmentation performance, it reduces required channel bandwidth by 50%–70%, while also decreasing storage overhead and computational complexity by 26% and 37%, respectively (Zhang et al., 27 Aug 2025).

1. Scope and design rationale

ProMSC-MIS is situated in multimodal semantic communication, where the communication objective is aligned with a downstream task rather than exact source recovery. In the formulation given for ProMSC-MIS, the downstream task is multi-spectral image segmentation, with RGB and thermal images treated as complementary sensing modalities. The motivating application domains explicitly include autonomous driving and nighttime surveillance, where wireless constraints, difficult lighting, and the complementary strengths of visible and thermal sensing make task-oriented transmission attractive (Zhang et al., 27 Aug 2025).

The framework is motivated by two limitations of prior practice. First, traditional communication systems emphasize accurate bit-level transmission with separate source and channel coding, which the description characterizes as inefficient in bandwidth- or computation-constrained settings and insufficiently aware of visual semantics. Second, much prior multimodal semantic communication work is described as focusing primarily on fusion strategies while neglecting the distinct contributions of each modality, with the consequence that learned representations may be redundant rather than diverse and complementary. ProMSC-MIS therefore defines four core objectives: efficient task-driven transmission, cross-modal complementarity, effective semantic fusion, and bandwidth and resource efficiency (Zhang et al., 27 Aug 2025).

A recurrent misconception in this area is that multimodal performance gains are determined chiefly by the fusion block. The ProMSC-MIS formulation explicitly rejects that narrow view by making unimodal feature diversity a pre-training target. In that sense, the system treats representation complementarity as a prerequisite for effective fusion rather than a by-product of fusion alone.

2. System composition and signal flow

The system operates on aligned RGB and thermal inputs. One description specifies synchronized inputs $\mathbf{x}^r \in \mathbb{R}^{H \times W \times 3}$ and $\mathbf{x}^t \in \mathbb{R}^{H \times W \times 1}$, processed at the transmitter by two unimodal semantic encoders, a semantic fusion module, and a transmission interface; the receiver reconstructs a segmentation output $$\mathbf{\hat{m}} \in \mathbb{R}^{H \times W \times N}$$ through a semantic decoder (Zhang et al., 25 Aug 2025).

The unimodal semantic encoders are described as sharing a design based on ResNet-152 with channel-wise and spatial attention modules, following the FEANet style while removing cross-modal interaction at the unimodal extraction stage. The RGB encoder $f_{RGB}(\cdot)$ extracts $\mathbf{y}_{RGB}^r$, and the thermal encoder $f_{THE}(\cdot)$ extracts $\mathbf{y}_{THE}^t$. These modality-specific semantic features are then fused into a compact semantic representation $\mathbf{z}_s$, which serves as the communication object rather than the original images (Zhang et al., 27 Aug 2025).

The transmission path is described in two compatible forms. In the digital path emphasized in the main framework description, $\mathbf{z}_s$ is quantized through a learnable bit generator into a bit sequence for transmission, and the receiver decodes the transmitted representation before semantic decoding. In a related formulation, ProMSC-MIS also supports Joint Source Channel Coding by transmitting $\mathbf{z}_s$ as real values, while Separate Source Channel Coding uses the learnable bit generator to digitize $\mathbf{x}^t \in \mathbb{R}^{H \times W \times 1}$0 into bits $\mathbf{x}^t \in \mathbb{R}^{H \times W \times 1}$1 (Zhang et al., 25 Aug 2025).

The semantic decoder is implemented as a stack of transpose convolution layers that upsamples the received representation, whether denoted $\mathbf{x}^t \in \mathbb{R}^{H \times W \times 1}$2 or decoded from $\mathbf{x}^t \in \mathbb{R}^{H \times W \times 1}$3, into a full-resolution pixel-wise segmentation mask. This transmitter–receiver decomposition makes the segmentation objective the organizing principle for both representation learning and transmission.

3. Prompt-based contrastive pre-training

A central novelty of ProMSC-MIS is its two-phase training strategy, in which unimodal semantic encoders are pre-trained using prompt learning and contrastive learning. The stated purpose is to force unimodal encoders to learn rich, diverse, and complementary semantic representations rather than overlapping ones. In this design, features or inputs from one modality serve as prompts for the encoder of the other modality, so that each encoder is exposed both to its native modality and to a transformed version of the other modality (Zhang et al., 27 Aug 2025).

The pre-training pipeline begins with channel alignment. RGB images are converted to grayscale to form $\mathbf{x}^t \in \mathbb{R}^{H \times W \times 1}$4, and thermal images are expanded to three channels to form $\mathbf{x}^t \in \mathbb{R}^{H \times W \times 1}$5, allowing both encoders to process cross-modal inputs. The encoders are then applied both to their own modality and to the prompted input: $\mathbf{x}^t \in \mathbb{R}^{H \times W \times 1}$6 These outputs are combined with same-modality features through learned projection modules: $\mathbf{x}^t \in \mathbb{R}^{H \times W \times 1}$7 where $\mathbf{x}^t \in \mathbb{R}^{H \times W \times 1}$8 and $\mathbf{x}^t \in \mathbb{R}^{H \times W \times 1}$9 are described as small neural networks or projection modules, implemented in one account as convolutional layers plus an MLP (Zhang et al., 25 Aug 2025).

The contrastive objective minimizes cosine similarity between the prompted projection outputs: $$\mathbf{\hat{m}} \in \mathbb{R}^{H \times W \times N}$$0 Because optimization seeks to reduce this value, the encoders are encouraged to move their learned representations apart, thereby increasing diversity. The intended effect is that each modality’s encoder becomes more sensitive to unique, task-relevant factors and better prepared for later fusion. The reported ablations indicate that the benefit of this pre-training is largest at low bits-per-pixel, which directly links complementarity learning to communication efficiency (Zhang et al., 27 Aug 2025).

4. Semantic fusion, enhancement, and bit generation

The semantic fusion module is a second major novelty. It is described as a two-stage structure that combines transformer-style cross-attention with squeeze-and-excitation enhancement. The first stage is a cross-attention module operating on the semantic feature tensors from RGB and thermal branches. Each modality is first refined by multi-head self-attention and feed-forward processing, for example

$$\mathbf{\hat{m}} \in \mathbb{R}^{H \times W \times N}$$1

followed by multi-head cross-attention to transfer information between modalities (Zhang et al., 27 Aug 2025).

Cross-modal interaction is then controlled by learnable refinement or gating. One formulation writes

$$\mathbf{\hat{m}} \in \mathbb{R}^{H \times W \times N}$$2

with an analogous branch for thermal features. Here $$\mathbf{\hat{m}} \in \mathbb{R}^{H \times W \times N}$$3 and $$\mathbf{\hat{m}} \in \mathbb{R}^{H \times W \times N}$$4 are learnable matrices used to adaptively reweight fusion and suppress harmful information. The outputs of the two branches are concatenated before the second fusion stage (Zhang et al., 25 Aug 2025).

The second stage, termed the fusion-enhancement module, alternates fusion blocks with squeeze-and-excitation networks. The fusion blocks use mini-inception structures to capture multi-scale context, while the SE blocks perform channel-wise global pooling followed by channel reweighting to emphasize informative feature channels. After three rounds of this processing, the system produces the final fused semantic representation $$\mathbf{\hat{m}} \in \mathbb{R}^{H \times W \times N}$$5 for transmission (Zhang et al., 27 Aug 2025).

The transition from floating-point semantic features to transmitted bits is handled by a learnable bit generator. Its output is a probability table $$\mathbf{\hat{m}} \in \mathbb{R}^{H \times W \times N}$$6, and training uses the Gumbel-Softmax trick to enable differentiable binary sampling. The motivation stated for this design is to avoid the performance degradation associated with fixed uniform quantization and to tailor digitization to the task and the bit budget (Zhang et al., 25 Aug 2025).

5. Optimization procedure and evaluation protocol

Training is organized into two stages. The first stage pre-trains the unimodal encoders using the contrastive cosine-similarity loss $$\mathbf{\hat{m}} \in \mathbb{R}^{H \times W \times N}$$7. The second stage performs end-to-end optimization of the communication and segmentation pipeline using a composite segmentation loss: $$\mathbf{\hat{m}} \in \mathbb{R}^{H \times W \times N}$$8 One formulation states that $$\mathbf{\hat{m}} \in \mathbb{R}^{H \times W \times N}$$9 is typically used. The Soft Cross-Entropy term is label-smoothed with $f_{RGB}(\cdot)$0, and $f_{RGB}(\cdot)$1 denotes the softmax probability for class $f_{RGB}(\cdot)$2 at pixel $f_{RGB}(\cdot)$3 (Zhang et al., 25 Aug 2025).

Experimental evaluation is conducted on the MFNet dataset, described as containing 1,569 pairs of RGB-T urban-scene images with day and night conditions and 8 object classes plus background; one account additionally notes an image size of $f_{RGB}(\cdot)$4 pixels. Performance is reported using mean Intersection over Union and mean Pixel Accuracy as functions of the bit-per-pixel budget. The digital transmission model uses a Binary Symmetric Channel with ideal error correction so that the study can focus on bandwidth rather than residual channel errors (Zhang et al., 27 Aug 2025).

The comparison set includes conventional image-transmission pipelines based on JPEG2000 or BPG combined with established segmentation networks such as MFNet, RTFNet, and FEANet. It also includes semantic-communication baselines: a multimodal DeepSC baseline with the same architecture as ProMSC-MIS but without pre-training, and unimodal DeepSC baselines for RGB-only and thermal-only transmission. The reported implementation environment includes an Intel Xeon CPU, an Nvidia L40 GPU, and PyTorch (Zhang et al., 27 Aug 2025).

6. Reported performance, ablations, and applications

The principal empirical claim is that ProMSC-MIS substantially outperforms conventional image transmission combined with state-of-the-art segmentation methods across channel-source compression levels. At the same segmentation performance, the framework is reported to reduce required channel bandwidth by 50%–70%; another summary states that it uses only 30%–50% of the bandwidth required by BPG-based pipelines and, in some cases, only $f_{RGB}(\cdot)$5 of that bandwidth. At $f_{RGB}(\cdot)$6 bpp, it achieves up to 3–4 times higher mIoU than JPEG2000-based methods and 45% higher mIoU than BPG-FEANet, while remaining operational at ultra-low bpp where conventional codecs fail completely (Zhang et al., 27 Aug 2025).

The data also report strong low-rate task performance. In the low-bpp regime, ProMSC-MIS is described as achieving more than 40% mIoU and more than 50% mAcc while outperforming all baselines. The system is further characterized as less sensitive to bandwidth fluctuation because it is optimized end to end for semantics rather than image fidelity (Zhang et al., 25 Aug 2025).

Resource-efficiency claims are also explicit. Relative to top-performing benchmarks such as RTFNet and FEANet, ProMSC-MIS achieves average reductions of 26% in parameters or storage and 37% in FLOPs, with improved inference latency. One detailed comparison reports 186.99M parameters, 212.34G FLOPs, and 46.95 ms inference latency for ProMSC-MIS; the corresponding figures are 254.51M, 337.46G, and 51.87 ms for RTFNet, and 255.21M, 337.47G, and 65.89 ms for FEANet. MFNet is much smaller, at 0.74M parameters and 8.42G FLOPs with 4.60 ms latency, but is described as far less accurate. The comparison further notes that baseline FLOPs exclude image coding time, which implies that the compute comparison is conservative with respect to ProMSC-MIS (Zhang et al., 25 Aug 2025).

Ablation studies provide the main internal validation. The gap between ProMSC-MIS and the multimodal DeepSC baseline without pre-training is reported to be largest at low bpp, supporting the role of prompt-based complementary feature learning. Multimodal models consistently outperform unimodal ones, and the modality analysis indicates that both RGB and thermal information are essential at low bitrates, while the contribution of RGB grows as bitrate increases because of its richer detail. The fusion ablations further show that both the SE networks and the cross-attention module contribute measurable gains to the fused representation (Zhang et al., 27 Aug 2025).

The stated application domains are autonomous driving and nighttime surveillance. In these settings, a plausible implication is that ProMSC-MIS is most relevant when communication links are constrained but segmentation quality must remain robust under illumination changes and cross-modal uncertainty.