LangGatedFusion in RIS-FUSION

• The paper introduces LangGatedFusion, a text-conditioned fusion module that integrates the semantics of a referring expression directly into the infrared-visible fusion backbone.
• It employs a combination of visual-to-text cross-attention, a soft spatial modality gate, and FiLM-like modulation to locally select which modality dominates at each image region.
• Joint optimization with referring image segmentation in RIS-FUSION boosts performance by over 11% in mIoU, enhancing targeted object localization.

Searching arXiv for the provided topic and related papers to ground the article in current literature. arXiv search query: "LangGatedFusion RIS-FUSION (Ma et al., 16 Sep 2025)" LangGatedFusion is the text-conditioning module introduced in "RIS-FUSION: Rethinking Text-Driven Infrared and Visible Image Fusion from the Perspective of Referring Image Segmentation" (Ma et al., 16 Sep 2025). In that framework, its function is to inject the semantics of a referring expression directly into an infrared-visible fusion backbone so that the fused image emphasizes the object described by the text rather than optimizing only generic perceptual quality. Architecturally, it combines visual-to-text cross-attention, a soft spatial modality gate, and FiLM-like modulation inside the deep stages of a dual-stream fusion encoder. Within RIS-FUSION, this module is part of a jointly optimized cascaded system for fusion and referring image segmentation (RIS), and the paper reports that the overall method outperforms existing methods by over $11\%$ in mIoU on the proposed benchmark (Ma et al., 16 Sep 2025).

1. Conceptual role and task definition

LangGatedFusion is defined around a specific diagnosis of text-driven infrared-visible image fusion: prior methods either refine a fused image after fusion is already finished, which limits the ability to reselect modality-specific content, or inject text into fusion without a well-aligned task to supervise whether the text actually helped. RIS-FUSION addresses this by coupling fusion to referring image segmentation, on the premise that both tasks share a common objective: highlighting the object referred to by the text. In this formulation, LangGatedFusion is the mechanism that makes the fusion backbone expression-aware at the feature level rather than only at the output level (Ma et al., 16 Sep 2025).

The underlying decision problem is region-specific. Visible input may be more informative for appearance, texture, and color attributes, whereas infrared input may be more informative for thermal saliency, low-light structure, or obscured targets. LangGatedFusion is introduced to make that selection locally and condition it on the referring expression. This gives the model a way to decide, for each region, whether visible or infrared evidence should dominate for the linguistic target. The same paper also introduces the MM-RIS benchmark, with $12.5\text{k}$ training and $3.5\text{k}$ testing triplets, each consisting of an infrared-visible image pair, a segmentation mask, and a referring expression, thereby aligning supervision and evaluation with the intended semantics of text-guided fusion (Ma et al., 16 Sep 2025).

2. Architectural placement within RIS-FUSION

RIS-FUSION is a cascaded two-stage framework composed of a Fusion Stage and an RIS Stage. The inputs are a visible image $I_{\text{vi}} \in \mathbb{R}^{3 \times H \times W}$, an infrared image $I_{\text{ir}} \in \mathbb{R}^{1 \times H \times W}$, and a text expression $T$. The visible image is converted to YCbCr, only the visible luminance $Y_{\text{vi}}$ is fused, the infrared image is treated as grayscale luminance $Y_{\text{ir}}$, and the visible chroma channels Cb/Cr are preserved for later recombination (Ma et al., 16 Sep 2025).

The fusion backbone is a dual-stream pyramid encoder followed by a U-Net decoder. One encoder processes visible luminance and the other processes infrared luminance; the two encoders do not share weights and operate at four resolution levels. Fusion is depth-dependent. At the first two scales, visible and infrared features are fused by element-wise addition. At the deeper two scales, LangGatedFusion replaces addition and performs text-conditioned fusion. After the decoder reconstructs fused luminance $Y_{\text{fuse}}$, it is recombined with visible Cb/Cr to produce the fused RGB image $\hat{I}_{\text{fuse}}$, which is then passed into the RIS stage, described as a Swin Transformer-based encoder-decoder. The same text embedding $12.5\text{k}$0 is also used in the RIS stage through Language-Guided Attention and decoder concatenation, so LangGatedFusion is one component of a broader language-conditioning strategy, but its specific role is in the deep fusion encoder itself (Ma et al., 16 Sep 2025).

3. Internal mechanism and mathematical formulation

At pyramid level $12.5\text{k}$1, LangGatedFusion receives a visible feature map $12.5\text{k}$2, an infrared feature map $12.5\text{k}$3, and a text embedding $12.5\text{k}$4. The text is encoded by a frozen BERT encoder. The visible and infrared features are first concatenated and passed to Language-Guided Attention (LGA) together with the token-level text embeddings. In LGA, the concatenated visual features are reshaped into pixel queries $12.5\text{k}$5, and the text embeddings are projected into keys and values $12.5\text{k}$6. The resulting context map is

$12.5\text{k}$7

This construction means that each spatial location queries the referring expression and receives a text-conditioned context vector aligned to image positions. The context map $12.5\text{k}$8 is then passed through convolutional heads that predict a soft gate $12.5\text{k}$9 and FiLM parameters $3.5\text{k}$0 and $3.5\text{k}$1:

$3.5\text{k}$2

$3.5\text{k}$3

The fusion rule is

$3.5\text{k}$4

The gate $3.5\text{k}$5 is a soft spatial selector between visible and infrared features. If $3.5\text{k}$6 is close to $3.5\text{k}$7, the fused feature at location $3.5\text{k}$8 relies more on visible input; if it is close to $3.5\text{k}$9, it relies more on infrared input. The equations reported in the paper indicate that $I_{\text{vi}} \in \mathbb{R}^{3 \times H \times W}$0, $I_{\text{vi}} \in \mathbb{R}^{3 \times H \times W}$1, and $I_{\text{vi}} \in \mathbb{R}^{3 \times H \times W}$2 are spatial maps of shape $I_{\text{vi}} \in \mathbb{R}^{3 \times H \times W}$3, so the modulation is spatially varying but shared across channels. Despite the name, LangGatedFusion in RIS-FUSION is therefore not a token-level expert router of the kind used in language-model ensembling; it is a text-conditioned spatial gate over deep visible and infrared feature maps (Ma et al., 16 Sep 2025).

4. Training objective and joint supervision

LangGatedFusion is trained as part of the entire RIS-FUSION system rather than with a separate stand-alone objective. The total loss combines segmentation and fusion terms:

$I_{\text{vi}} \in \mathbb{R}^{3 \times H \times W}$4

The segmentation loss is Dice loss on the predicted binary mask:

$I_{\text{vi}} \in \mathbb{R}^{3 \times H \times W}$5

where $I_{\text{vi}} \in \mathbb{R}^{3 \times H \times W}$6 is the predicted mask and $I_{\text{vi}} \in \mathbb{R}^{3 \times H \times W}$7 is the ground-truth mask. The fusion loss is defined on luminance and includes a visible-image SSIM term, visible and infrared MSE terms, an infrared Sobel term, and gradient consistency terms using $I_{\text{vi}} \in \mathbb{R}^{3 \times H \times W}$8 and $I_{\text{vi}} \in \mathbb{R}^{3 \times H \times W}$9. The reported weights are $I_{\text{ir}} \in \mathbb{R}^{1 \times H \times W}$0, $I_{\text{ir}} \in \mathbb{R}^{1 \times H \times W}$1, $I_{\text{ir}} \in \mathbb{R}^{1 \times H \times W}$2, $I_{\text{ir}} \in \mathbb{R}^{1 \times H \times W}$3, and $I_{\text{ir}} \in \mathbb{R}^{1 \times H \times W}$4 (Ma et al., 16 Sep 2025).

A central training property is that the fused image $I_{\text{ir}} \in \mathbb{R}^{1 \times H \times W}$5 is fed into the RIS network without gradient detachment. Consequently, segmentation loss back-propagates into the fusion module, including LangGatedFusion. This gives the gate and FiLM parameters semantic pressure from the downstream RIS task rather than only low-level image-reconstruction pressure. The intended effect is that the fusion module learns not merely to produce perceptually plausible fused images, but to emphasize the target region in a way that improves binary mask prediction for the referring expression (Ma et al., 16 Sep 2025).

5. Empirical behavior and ablation evidence

On the main benchmark, RIS-FUSION with text enabled reports $I_{\text{ir}} \in \mathbb{R}^{1 \times H \times W}$6, $I_{\text{ir}} \in \mathbb{R}^{1 \times H \times W}$7, $I_{\text{ir}} \in \mathbb{R}^{1 \times H \times W}$8, $I_{\text{ir}} \in \mathbb{R}^{1 \times H \times W}$9, $T$0, and $T$1. The paper states that this exceeds prior methods by over $T$2 in mIoU, and in the tabulated comparison it improves over the best previous listed baseline, OmniFuse+RIS$T$3, from $T$4 to $T$5 mIoU. The comparison between RIS-FUSION without referring text and with referring text is also direct: $T$6 rises from $T$7 to $T$8, and $T$9 rises from $Y_{\text{vi}}$0 to $Y_{\text{vi}}$1, indicating that the language-conditioned fusion pipeline contributes materially to target localization rather than only to generic fusion quality (Ma et al., 16 Sep 2025).

The ablation study isolates both LangGatedFusion and joint optimization. Without LangGatedFusion and without joint optimization, the model reports $Y_{\text{vi}}$2 mIoU and $Y_{\text{vi}}$3 $Y_{\text{vi}}$4. Adding LangGatedFusion alone raises these values to $Y_{\text{vi}}$5 and $Y_{\text{vi}}$6. Using joint optimization without LangGatedFusion gives $Y_{\text{vi}}$7 mIoU and $Y_{\text{vi}}$8 $Y_{\text{vi}}$9. Combining LangGatedFusion with joint optimization yields the best result, $Y_{\text{ir}}$0 mIoU and $Y_{\text{ir}}$1 $Y_{\text{ir}}$2. This pattern indicates that LangGatedFusion contributes benefits distinct from the RIS-supervised cascade itself: joint optimization globally aligns fusion with RIS supervision, whereas LangGatedFusion provides direct local text conditioning inside the fusion backbone. The paper also reports qualitative evidence that, for a prompt such as “two color cones on the left side of the road,” RIS-FUSION suppresses over-exposure, highlights the target cones, and avoids distraction by other salient objects (Ma et al., 16 Sep 2025).

6. Relation to broader gated-fusion research

LangGatedFusion belongs to a broader class of gated fusion mechanisms, but its specific form is unusually narrow and task-aligned. A useful contrast is "GateFusion: Hierarchical Gated Cross-Modal Fusion for Active Speaker Detection" (Wang et al., 17 Dec 2025), which performs progressive multi-depth fusion by projecting hidden states from one modality into another through bimodally conditioned residual gates. GateFusion is hierarchical and bidirectional across encoder depths, whereas LangGatedFusion is applied only at the deeper two scales of the fusion encoder and uses language-guided attention to derive a spatial modality gate.

A second comparison is TGIF, "Text-Guided Layer Fusion Mitigates Hallucination in Multimodal LLMs" (Lin et al., 6 Jan 2026). TGIF treats frozen vision-encoder layers as depth-wise experts and predicts a prompt-dependent softmax over layers using a pooled text embedding. That is a global per-example router over encoder depth. LangGatedFusion, by contrast, predicts a spatial gate over visible and infrared features after local visual-to-text alignment. Both are text-guided fusion modules, but the conditioning granularity and fusion axis differ: TGIF gates layers, while LangGatedFusion gates modalities across image locations.

In generative LLM adaptation, LoRA-Flow introduces dynamic LoRA fusion with a layer-wise gate conditioned on the current hidden state, producing token- and layer-dependent mixtures of pretrained LoRA outputs during decoding (Wang et al., 2024). Adapter Fusion for multilingual Text2Cypher uses a fusion MLP with dynamic gating at the logit level, but the gating is sequence-level and output-level rather than spatial or layerwise within the network (Ozsoy, 22 Jan 2026). DLLG extends this line to token-level logit fusion of frozen LLM experts during autoregressive generation (Li et al., 3 Jun 2026). These methods suggest that “language-gated fusion” can refer to several distinct granularities—spatial feature fusion, layer fusion, adapter fusion, and logit fusion—whereas LangGatedFusion in RIS-FUSION is specifically a text-conditioned spatial gate over multimodal visual features.

The narrower study "Fusion Matters: Length-Aware Analysis of Positional-Encoding Fusion in Transformers" (Hallam et al., 9 Jan 2026) provides a complementary lesson: even lightweight learnable gates can outperform fixed additive fusion when the relative value of two signals varies across positions. This suggests, by analogy, why RIS-FUSION benefits from replacing static visible-infrared combination with a learnable text-conditioned gate. A plausible implication is that LangGatedFusion should be understood less as an isolated module name and more as one instance of a general design principle: when the importance of competing sources changes across locations, examples, or decoding steps, explicit gating is preferable to unconditional addition.

7. Limitations and unresolved design questions

The conceptual mechanism of LangGatedFusion is clear, but several implementation details are not specified in the paper. The accessible description does not state the exact BERT variant, the exact channel dimension $Y_{\text{ir}}$3 at each level, the exact value of the FiLM strength $Y_{\text{ir}}$4, the precise projection layers used to form $Y_{\text{ir}}$5, $Y_{\text{ir}}$6, and $Y_{\text{ir}}$7, whether LGA is single-head or multi-head, or the kernel sizes and other low-level details of $Y_{\text{ir}}$8 and $Y_{\text{ir}}$9. The paper also does not specify whether $Y_{\text{fuse}}$0 and $Y_{\text{fuse}}$1 remain single-channel in code or become multi-channel in practice. These omissions make the module conceptually reproducible but not fully specified at engineering level (Ma et al., 16 Sep 2025).

The gating granularity is another limitation. The reported equations imply that $Y_{\text{fuse}}$2, $Y_{\text{fuse}}$3, and $Y_{\text{fuse}}$4 are spatial maps of shape $Y_{\text{fuse}}$5, so the gate is spatial but not explicitly channel-wise. This is simple and interpretable, but it may restrict feature-subspace selectivity. The design also injects text only at the deeper two scales, leaving the first two scales to element-wise addition; the paper motivates this choice by assigning text a primarily semantic role, but it does not test whether low-level text-conditioned fusion might help with small or thin referred objects. Internal interpretability is likewise limited: the paper reports no explicit visualizations of gate maps $Y_{\text{fuse}}$6, FiLM fields $Y_{\text{fuse}}$7, or token-level attention maps. Finally, the ablation study compares the presence and absence of LangGatedFusion, but not FiLM-only, gate-only, or channel-wise alternatives, so the relative contribution of each internal ingredient remains unresolved (Ma et al., 16 Sep 2025).