FMISeg: Frequency-domain Multi-modal Segmentation

• The paper introduces FMISeg as a language-guided model that decomposes images using a 2D DWT, capturing both large-scale context (LF) and fine details (HF).
• The FFBI module uses bidirectional multi-head cross-attention between dual-branch features to effectively merge low- and high-frequency information, improving segmentation boundaries.
• The LFFI module injects clinical text semantics into the decoder, filtering irrelevant features and enhancing delineation of pulmonary infection lesions.

Frequency-domain Multi-modal Interaction Model (FMISeg) is a language-guided medical image segmentation architecture that establishes interaction between linguistic features and frequency-domain visual features in the decoder, while using wavelet-based decomposition to obtain complementary low-frequency and high-frequency visual inputs (Yu et al., 24 Sep 2025). In the formulation reported for pulmonary infection segmentation, FMISeg is a late fusion model: a dual-branch visual encoder processes low-frequency and high-frequency images derived from a 2D discrete wavelet transform, a Frequency-domain Feature Bidirectional Interaction module enhances visual representation by exchanging information between the two frequency branches, and a Language-guided Frequency-domain Feature Interaction module injects clinical text semantics into the decoder to suppress semantically irrelevant visual features (Yu et al., 24 Sep 2025). Within the broader literature, FMISeg belongs to a line of work that treats frequency-domain information not as a preprocessing artifact but as a complementary representational space for multimodal reasoning, as also seen in manipulation analysis, audio-visual segmentation, and frequency-oriented multimodal localization frameworks (Liu et al., 2023, Shen et al., 23 Sep 2025, Liu et al., 21 Feb 2026).

1. Conceptual basis and problem setting

FMISeg is motivated by two claims stated explicitly in the source literature. First, pulmonary infection lesions in CT or X-ray exhibit “complex morphology,” including “small, scattered foci, fuzzy/low-contrast borders, heterogeneous texture,” which makes purely spatial-domain feature extraction insufficient for simultaneously modeling global context and fine structural cues (Yu et al., 24 Sep 2025). Second, clinical text reports provide semantic priors such as lesion location, shape, extent, and contextual descriptors, which can disambiguate visual ambiguity and suppress irrelevant anatomy (Yu et al., 24 Sep 2025).

The model is positioned against prior language-guided medical image segmentation methods that “mostly fuse text and image in the spatial domain” and “often rely on unidirectional cross-attention” (Yu et al., 24 Sep 2025). The reported limitations are threefold: the semantic gap between image textures and linguistic descriptions, inadequate suppression of semantically irrelevant visual features in anatomically complex backgrounds, and weak handling of “small/low-contrast lesions whose boundaries are easily confused with normal structures” (Yu et al., 24 Sep 2025). FMISeg addresses these issues by decomposing the image into low-frequency and high-frequency components and then performing bidirectional interaction both within the visual frequency branches and between visual and textual representations (Yu et al., 24 Sep 2025).

The underlying rationale is consistent with other frequency-aware multimodal models. In UFAFormer for detecting and grounding multi-modal manipulation, the frequency domain is introduced as “a complementary visual viewpoint” that preserves spatial structure while exposing manipulation artifacts not easily captured from RGB alone (Liu et al., 2023). In frequency-aware audio-visual segmentation, the literature similarly argues that multimodal systems may fail when one modality’s high frequencies are predominantly noise while another modality’s high frequencies encode structurally important details (Shen et al., 23 Sep 2025). This suggests that FMISeg is part of a broader methodological shift toward modality-aware spectral decomposition rather than generic cross-modal fusion.

2. Frequency-domain representation and dual-branch visual encoding

FMISeg uses a 2D discrete wavelet transform to decompose the raw image into low-frequency and high-frequency images (Yu et al., 24 Sep 2025). The paper follows XNet-style wavelet decomposition and states that the transform type is DWT rather than FFT or DCT (Yu et al., 24 Sep 2025). A standard single-level separable 2D DWT formulation is provided in the source to clarify the process:

• $$S_{LL}[m,n] = \sum_{x=0}^{M-1} \sum_{y=0}^{N-1} f(x,y)\, h_L[2m-x]\, h_L[2n-y]$$
• $$S_{LH}[m,n] = \sum_{x=0}^{M-1} \sum_{y=0}^{N-1} f(x,y)\, h_L[2m-x]\, h_H[2n-y]$$
• $$S_{HL}[m,n] = \sum_{x=0}^{M-1} \sum_{y=0}^{N-1} f(x,y)\, h_H[2m-x]\, h_L[2n-y]$$
• $$S_{HH}[m,n] = \sum_{x=0}^{M-1} \sum_{y=0}^{N-1} f(x,y)\, h_H[2m-x]\, h_H[2n-y]$$

In this formulation, the low-frequency image is defined as $I_L = S_{LL}$, while the high-frequency image $I_H$ is constructed from the high-frequency subbands; the exact stacking scheme is not specified in the paper, which states only that it follows XNet-style HF/LF decomposition (Yu et al., 24 Sep 2025). That omission matters for reproducibility, and the paper itself lists the unspecified wavelet type and decomposition level as a limitation (Yu et al., 24 Sep 2025).

The visual encoder is dual-branch. A ConvNeXt-Tiny backbone processes the LF image and another ConvNeXt-Tiny backbone processes the HF image (Yu et al., 24 Sep 2025). Both branches produce multi-scale features at downsampling rates 4, 8, 16, and 32: $F_m^1 \in \mathbb{R}^{H/4 \times W/4 \times C_1}$, $F_m^2 \in \mathbb{R}^{H/8 \times W/8 \times C_2}$, $F_m^3 \in \mathbb{R}^{H/16 \times W/16 \times C_3}$, $F_m^4 \in \mathbb{R}^{H/32 \times W/32 \times C_4}$, for $$S_{LH}[m,n] = \sum_{x=0}^{M-1} \sum_{y=0}^{N-1} f(x,y)\, h_L[2m-x]\, h_H[2n-y]$$0 (Yu et al., 24 Sep 2025).

The representational division is explicit. Low-frequency components emphasize “large-scale structure and context,” whereas high-frequency components preserve “fine textural detail and boundaries” (Yu et al., 24 Sep 2025). In the intended medical setting, this complementarity is directly tied to segmentation quality: LF supports coarse lesion localization and organ-level context, while HF supports lesion edge disambiguation and the recovery of subtle structures (Yu et al., 24 Sep 2025).

3. Visual-frequency interaction through FFBI

The Frequency-domain Feature Bidirectional Interaction (FFBI) module is the visual fusion component of FMISeg (Yu et al., 24 Sep 2025). It operates at the last encoder stage and exchanges information between the deepest LF and HF feature maps via bidirectional multi-head cross-attention (Yu et al., 24 Sep 2025).

The inputs are:

• $$S_{LH}[m,n] = \sum_{x=0}^{M-1} \sum_{y=0}^{N-1} f(x,y)\, h_L[2m-x]\, h_H[2n-y]$$1
• $$S_{LH}[m,n] = \sum_{x=0}^{M-1} \sum_{y=0}^{N-1} f(x,y)\, h_L[2m-x]\, h_H[2n-y]$$2

The paper gives the FFBI equations as:

• $$S_{LH}[m,n] = \sum_{x=0}^{M-1} \sum_{y=0}^{N-1} f(x,y)\, h_L[2m-x]\, h_H[2n-y]$$3
• $$S_{LH}[m,n] = \sum_{x=0}^{M-1} \sum_{y=0}^{N-1} f(x,y)\, h_L[2m-x]\, h_H[2n-y]$$4 (Yu et al., 24 Sep 2025)

The multi-head cross-attention operator is described using standard scaled dot-product attention. For head $$S_{LH}[m,n] = \sum_{x=0}^{M-1} \sum_{y=0}^{N-1} f(x,y)\, h_L[2m-x]\, h_H[2n-y]$$5,

• $$S_{LH}[m,n] = \sum_{x=0}^{M-1} \sum_{y=0}^{N-1} f(x,y)\, h_L[2m-x]\, h_H[2n-y]$$6
• $$S_{LH}[m,n] = \sum_{x=0}^{M-1} \sum_{y=0}^{N-1} f(x,y)\, h_L[2m-x]\, h_H[2n-y]$$7
• $$S_{LH}[m,n] = \sum_{x=0}^{M-1} \sum_{y=0}^{N-1} f(x,y)\, h_L[2m-x]\, h_H[2n-y]$$8
• $$S_{LH}[m,n] = \sum_{x=0}^{M-1} \sum_{y=0}^{N-1} f(x,y)\, h_L[2m-x]\, h_H[2n-y]$$9
• $$S_{HL}[m,n] = \sum_{x=0}^{M-1} \sum_{y=0}^{N-1} f(x,y)\, h_H[2m-x]\, h_L[2n-y]$$0 (Yu et al., 24 Sep 2025)

The purpose of FFBI is asymmetrical but bidirectional. HF receives semantic and global guidance from LF, which helps “reject HF noise and sharpen boundaries in context,” while LF receives fine-detail cues from HF, improving localization and border accuracy (Yu et al., 24 Sep 2025). Residual connections and layer normalization preserve the original signals and stabilize fusion (Yu et al., 24 Sep 2025).

The ablation results reported for FMISeg support the claim that HF and LF are complementary. On QaTa-COV19 and MosMedData+, the two-branch model with FFBI outperforms single-branch raw-image, HF-only, LF-only, and simple two-branch concatenation baselines (Yu et al., 24 Sep 2025). The source reports the following Dice/mIoU values:

Configuration	QaTa-COV19	MosMedData+
Single-branch raw image	89.86 / 81.72	78.21 / 64.17
Single-branch HF only	88.75 / 80.15	77.16 / 63.04
Single-branch LF only	89.54 / 81.23	77.89 / 63.69
Two-branch concat(HF, LF)	90.61 / 82.93	78.65 / 64.88
Two-branch with FFBI	91.21 / 83.84	79.30 / 65.71

These results are used in the paper to conclude that “HF/LF are complementary” and that FFBI’s bidirectional cross-attention is “superior to simple concatenation” (Yu et al., 24 Sep 2025).

4. Language-guided decoding through LFFI

FMISeg’s textual pathway uses CXR-BERT to encode clinical text into word-level features $$S_{HL}[m,n] = \sum_{x=0}^{M-1} \sum_{y=0}^{N-1} f(x,y)\, h_H[2m-x]\, h_L[2n-y]$$1 (Yu et al., 24 Sep 2025). The paper specifies that word-level features are used and that CLS pooling is not used (Yu et al., 24 Sep 2025). This textual representation is injected late, in the decoder, through the Language-guided Frequency-domain Feature Interaction (LFFI) module (Yu et al., 24 Sep 2025).

At a decoder stage, LFFI receives a visual feature map $$S_{HL}[m,n] = \sum_{x=0}^{M-1} \sum_{y=0}^{N-1} f(x,y)\, h_H[2m-x]\, h_L[2n-y]$$2 for $$S_{HL}[m,n] = \sum_{x=0}^{M-1} \sum_{y=0}^{N-1} f(x,y)\, h_H[2m-x]\, h_L[2n-y]$$3 and the word-level textual features $$S_{HL}[m,n] = \sum_{x=0}^{M-1} \sum_{y=0}^{N-1} f(x,y)\, h_H[2m-x]\, h_L[2n-y]$$4 (Yu et al., 24 Sep 2025). It first performs bidirectional cross-attention:

• $$S_{HL}[m,n] = \sum_{x=0}^{M-1} \sum_{y=0}^{N-1} f(x,y)\, h_H[2m-x]\, h_L[2n-y]$$5
• $$S_{HL}[m,n] = \sum_{x=0}^{M-1} \sum_{y=0}^{N-1} f(x,y)\, h_H[2m-x]\, h_L[2n-y]$$6 (Yu et al., 24 Sep 2025)

It then computes an interaction tensor and a learned gate:

• $$S_{HL}[m,n] = \sum_{x=0}^{M-1} \sum_{y=0}^{N-1} f(x,y)\, h_H[2m-x]\, h_L[2n-y]$$7
• $$S_{HL}[m,n] = \sum_{x=0}^{M-1} \sum_{y=0}^{N-1} f(x,y)\, h_H[2m-x]\, h_L[2n-y]$$8
• $$S_{HL}[m,n] = \sum_{x=0}^{M-1} \sum_{y=0}^{N-1} f(x,y)\, h_H[2m-x]\, h_L[2n-y]$$9 (Yu et al., 24 Sep 2025)

The paper interprets this gating mechanism as a “semantically irrelevant filter” that suppresses text-irrelevant visual content (Yu et al., 24 Sep 2025). Cross-attention alone may import noise and lacks positional granularity; the gate therefore uses the interaction between visual and textual attended features to generate spatially varying, channel-wise reweighting (Yu et al., 24 Sep 2025).

The design is explicitly late fusion. Frequency fusion occurs at the deepest encoder stage through FFBI, while text fusion occurs progressively through LFFI in the decoder (Yu et al., 24 Sep 2025). The paper reports that adding more LFFI layers from high to low decoder levels yields incremental performance improvements (Yu et al., 24 Sep 2025). The corresponding ablation on QaTa-COV19 and MosMedData+ shows a monotonic progression from “No text” to “4 layers”:

LFFI depth	QaTa-COV19	MosMedData+
No text	87.63 / 78.13	76.45 / 62.87
1 LFFI layer	90.64 / 82.98	78.71 / 64.93
2 layers	90.86 / 83.26	78.97 / 65.25
3 layers	91.06 / 83.63	79.16 / 65.48
4 layers	91.21 / 83.84	79.30 / 65.71

The paper concludes that “progressive late fusion of text improves segmentation” and that “more LFFI layers yield incremental gains” (Yu et al., 24 Sep 2025).

This decoder-centric language integration parallels a broader design choice in multimodal segmentation. In UFAFormer, a unified decoder with symmetric cross-modal interaction modules is used to align visual and textual forgery cues (Liu et al., 2023). In FAVS, cross-modal consistency is enforced progressively after frequency-domain decomposition and recomposition (Shen et al., 23 Sep 2025). FMISeg differs in its specific domain and architectural details, but the common principle is that language or another complementary modality should modulate frequency-enriched visual features after sufficient visual abstraction has been established.

5. Training setup, datasets, and empirical performance

FMISeg is evaluated on two datasets. QaTa-COV19 contains “9,258 CXR images with lesion annotations and textual descriptions,” split into 5,716 training, 1,429 validation, and 2,113 test images (Yu et al., 24 Sep 2025). MosMedData+ contains “2,729 COVID-19 CT slices with binary masks,” split into 2,183 training, 273 validation, and 273 test images following prior language-guided medical image segmentation work (Yu et al., 24 Sep 2025).

The training setup reported in the paper is precise. Input resolution is $$S_{HH}[m,n] = \sum_{x=0}^{M-1} \sum_{y=0}^{N-1} f(x,y)\, h_H[2m-x]\, h_H[2n-y]$$0 for fair comparison (Yu et al., 24 Sep 2025). The framework is implemented in PyTorch, uses ConvNeXt-Tiny for both LF and HF branches, AdamW optimization, an initial learning rate of $$S_{HH}[m,n] = \sum_{x=0}^{M-1} \sum_{y=0}^{N-1} f(x,y)\, h_H[2m-x]\, h_H[2n-y]$$1 with cosine annealing down to $$S_{HH}[m,n] = \sum_{x=0}^{M-1} \sum_{y=0}^{N-1} f(x,y)\, h_H[2m-x]\, h_H[2n-y]$$2, batch size 32, hidden dimension 768 in the interaction modules, and NVIDIA RTX 3090 hardware (Yu et al., 24 Sep 2025). The paper does not detail intensity normalization or augmentation policies (Yu et al., 24 Sep 2025).

The objective function combines Dice loss and binary cross-entropy (Yu et al., 24 Sep 2025). The source gives:

• $$S_{HH}[m,n] = \sum_{x=0}^{M-1} \sum_{y=0}^{N-1} f(x,y)\, h_H[2m-x]\, h_H[2n-y]$$3
• $$S_{HH}[m,n] = \sum_{x=0}^{M-1} \sum_{y=0}^{N-1} f(x,y)\, h_H[2m-x]\, h_H[2n-y]$$4 (Yu et al., 24 Sep 2025)

No auxiliary frequency-regularization or cross-modal alignment losses are introduced in FMISeg itself (Yu et al., 24 Sep 2025). That is notable because related work such as FOCA includes a composite objective with mask, classification, text-generation, and contrastive terms (Liu et al., 21 Feb 2026), while the FMISeg formulation remains limited to dense prediction supervision.

The main quantitative comparison reported by the paper is as follows:

Method	QaTa-COV19 Dice / mIoU	MosMedData+ Dice / mIoU
FMISeg	91.21 / 83.84	79.30 / 65.71
nnUNet	80.42 / 70.81	72.59 / 60.36
LGA	84.65 / 76.23	—
TGCAM	90.60 / 82.81	77.82 / 63.69

The paper states that FMISeg achieves gains of “+10.79 Dice, +13.03 mIoU” over nnUNet on QaTa-COV19 and “+6.71 Dice, +5.35 mIoU” on MosMedData+ (Yu et al., 24 Sep 2025). Against TGCAM, the reported improvements are “+0.61 Dice, +1.03 mIoU” on QaTa-COV19 and “+1.48 Dice, +2.02 mIoU” on MosMedData+ (Yu et al., 24 Sep 2025). The qualitative comparison is summarized by the claim that FMISeg produces “cleaner boundaries, fewer missed small lesions, and better suppression of false positives in complex regions” than uni-modal and language-guided baselines (Yu et al., 24 Sep 2025).

6. Relation to adjacent frequency-aware multimodal models

Although FMISeg is a medical image segmentation model, its design sits within a larger family of frequency-aware multimodal architectures that differ in modality pairings, task definitions, and fusion operators.

UFAFormer addresses “Detecting and Grounding Multi-modal Manipulation” by introducing a discrete wavelet transform-based frequency encoder, a forgery-aware mutual module, and a unified decoder for image-text reasoning (Liu et al., 2023). Its detailed architecture includes intra-band and inter-band self-attentions over DWT sub-bands, mutual cross-attention between RGB and frequency features, and symmetric text-visual cross-modal interaction modules (Liu et al., 2023). The paper’s adaptation notes explicitly state that these design choices “directly inform a Frequency-domain Multi-modal Interaction Segmentation model (FMISeg),” namely by retaining the DWT frequency encoder and mutual alignment machinery while replacing detection heads with a segmentation head (Liu et al., 2023). This is not the same FMISeg as the medical segmentation model of 2025, but it establishes a second usage of the term as an adaptation blueprint for dense localization of manipulated regions.

FAVS reformulates audio-visual segmentation as a “frequency-domain decomposition and recomposition problem” (Shen et al., 23 Sep 2025). Its Frequency-Domain Enhanced Decomposer separates features into high-, mid-, low-frequency bands and a residual via residual-based iterative decomposition, while the Synergistic Cross-Modal Consistency module uses a mixture-of-experts architecture with dynamic expert routing to align audio and visual features (Shen et al., 23 Sep 2025). The model reports state-of-the-art performance on AVSBench subsets and attributes gains to better handling of the modality-specific contradiction that “audio high-frequency signals” are often noisy while “visual high-frequency signals” are structurally informative (Shen et al., 23 Sep 2025). Relative to FMISeg, this framework broadens the scope of frequency-aware multimodal interaction from language-guided medical imaging to audio-visual dense prediction.

FOCA extends frequency-aware multimodality into detection, localization, and explanation using a multimodal LLM (Liu et al., 21 Feb 2026). It uses DWT to extract frequency cues, with $$S_{HH}[m,n] = \sum_{x=0}^{M-1} \sum_{y=0}^{N-1} f(x,y)\, h_H[2m-x]\, h_H[2n-y]$$5 serving as the query in a Frequency Attention Fusion cross-attention:

$$S_{HH}[m,n] = \sum_{x=0}^{M-1} \sum_{y=0}^{N-1} f(x,y)\, h_H[2m-x]\, h_H[2n-y]$$6

and residual fusion

$$S_{HH}[m,n] = \sum_{x=0}^{M-1} \sum_{y=0}^{N-1} f(x,y)\, h_H[2m-x]\, h_H[2n-y]$$7

(Liu et al., 21 Feb 2026). FOCA’s use of explicit segmentation tokens, a frozen vision encoder, and joint mask-generation and explanation objectives suggests a different trajectory for frequency-aware multimodal interaction: coupling spectral guidance with instruction-conditioned reasoning and interpretability (Liu et al., 21 Feb 2026).

Taken together, these systems show three recurring principles. First, frequency-domain representations are used to preserve information that may be weakly expressed in the spatial domain. Second, multimodal fusion is usually not simple concatenation; it is mediated by cross-attention, mutual alignment, or expert routing. Third, dense prediction quality is improved when frequency information remains spatially aligned, as in DWT-based formulations (Liu et al., 2023, Shen et al., 23 Sep 2025, Yu et al., 24 Sep 2025, Liu et al., 21 Feb 2026). This suggests that FMISeg is best understood not as an isolated architecture but as one instance of a broader methodological pattern.

7. Limitations, misconceptions, and plausible directions

A common misconception is that FMISeg is merely a conventional language-guided segmentation model with an added wavelet preprocessing step. The reported architecture contradicts that interpretation. The frequency decomposition is structurally embedded into the model through a dual-branch encoder, FFBI bidirectional interaction, and LFFI-based language-guided filtering in the decoder (Yu et al., 24 Sep 2025). The performance gap between simple HF/LF concatenation and FFBI also indicates that the claimed benefit is not attributed to decomposition alone but to the interaction mechanism built on top of it (Yu et al., 24 Sep 2025).

Another possible misconception is that “frequency-domain” in FMISeg denotes a generic spectral transform. The paper explicitly notes that FMISeg does not use FFT or DCT and instead adopts DWT to form LF and HF images (Yu et al., 24 Sep 2025). The reason, stated indirectly through the architectural design, is that DWT preserves spatial structure in a way that can be exploited by encoder-decoder segmentation pipelines. This is consistent with UFAFormer’s use of DWT to preserve spatial layout while exposing texture and edge anomalies (Liu et al., 2023).

The limitations reported for FMISeg are concrete. The wavelet type and decomposition level are “not explicitly stated,” which affects reproducibility (Yu et al., 24 Sep 2025). The experiments focus on “2D slices (CXR, CT slices),” so extension to 3D volumetric segmentation would require “3D wavelet transforms and 3D attention, with memory considerations” (Yu et al., 24 Sep 2025). Only English chest-report domain text via CXR-BERT is used, and sensitivity to report quality or prompt formulation is not analyzed (Yu et al., 24 Sep 2025). The paper also notes that robustness to noisy or incomplete reports is not formally tested (Yu et al., 24 Sep 2025).

Several future directions are identified in the surrounding literature rather than established as facts for FMISeg itself. The medical FMISeg paper points to broader anatomy, multi-disease settings, and richer clinical narratives as natural extensions (Yu et al., 24 Sep 2025). The FAVS work discusses adaptive band selection, learnable filter banks, multi-scale frequency fusion, and explicit frequency-domain alignment losses as possible strengthenings of a general FMISeg design (Shen et al., 23 Sep 2025). FOCA, meanwhile, suggests that frequency-aware segmentation can be coupled to multimodal LLMs for explanation and cross-domain interpretability (Liu et al., 21 Feb 2026). These are not part of FMISeg as reported, but they indicate plausible directions in which the architecture class may evolve.

In that sense, FMISeg occupies a specific and technically well-defined point in the development of frequency-aware multimodal segmentation: it operationalizes wavelet-domain decomposition, bidirectional LF/HF interaction, and decoder-stage language guidance for pulmonary infection lesion segmentation, while leaving open questions about transform specification, volumetric generalization, textual robustness, and more explicit cross-modal alignment objectives (Yu et al., 24 Sep 2025).