---
title: 'FMISeg: Frequency-domain Multi-modal Segmentation'
url: https://www.emergentmind.com/topics/frequency-domain-multi-modal-interaction-model-fmiseg
type: topic
---

# FMISeg: Frequency-domain Multi-modal Segmentation

Frequency-domain Multi-modal Interaction Model (FMISeg) is a language-guided medical image segmentation architecture that establishes interaction between linguistic features and frequency-domain visual features in the decoder, while using wavelet-based decomposition to obtain complementary low-frequency and high-frequency visual inputs [2509.19719]. In the formulation reported for pulmonary infection segmentation, FMISeg is a late fusion model: a dual-branch visual encoder processes low-frequency and high-frequency images derived from a 2D discrete wavelet transform, a Frequency-domain Feature Bidirectional Interaction module enhances visual representation by exchanging information between the two frequency branches, and a Language-guided Frequency-domain Feature Interaction module injects clinical text semantics into the decoder to suppress semantically irrelevant visual features [2509.19719]. Within the broader literature, FMISeg belongs to a line of work that treats frequency-domain information not as a preprocessing artifact but as a complementary representational space for multimodal reasoning, as also seen in manipulation analysis, audio-visual segmentation, and frequency-oriented multimodal localization frameworks [2309.09667] [2509.18912] [2602.18880].

## 1. Conceptual basis and problem setting

FMISeg is motivated by two claims stated explicitly in the source literature. First, pulmonary infection lesions in CT or X-ray exhibit “complex morphology,” including “small, scattered foci, fuzzy/low-contrast borders, heterogeneous texture,” which makes purely spatial-domain feature extraction insufficient for simultaneously modeling global context and fine structural cues [2509.19719]. Second, clinical text reports provide semantic priors such as lesion location, shape, extent, and contextual descriptors, which can disambiguate visual ambiguity and suppress irrelevant anatomy [2509.19719].

The model is positioned against prior language-guided medical image segmentation methods that “mostly fuse text and image in the spatial domain” and “often rely on unidirectional cross-attention” [2509.19719]. The reported limitations are threefold: the semantic gap between image textures and linguistic descriptions, inadequate suppression of semantically irrelevant visual features in anatomically complex backgrounds, and weak handling of “small/low-contrast lesions whose boundaries are easily confused with normal structures” [2509.19719]. FMISeg addresses these issues by decomposing the image into low-frequency and high-frequency components and then performing bidirectional interaction both within the visual frequency branches and between visual and textual representations [2509.19719].

The underlying rationale is consistent with other frequency-aware multimodal models. In UFAFormer for detecting and grounding multi-modal manipulation, the frequency domain is introduced as “a complementary visual viewpoint” that preserves spatial structure while exposing manipulation artifacts not easily captured from RGB alone [2309.09667]. In frequency-aware audio-visual segmentation, the literature similarly argues that multimodal systems may fail when one modality’s high frequencies are predominantly noise while another modality’s high frequencies encode structurally important details [2509.18912]. This suggests that FMISeg is part of a broader methodological shift toward modality-aware spectral decomposition rather than generic cross-modal fusion.

## 2. Frequency-domain representation and dual-branch visual encoding

FMISeg uses a 2D discrete wavelet transform to decompose the raw image into low-frequency and high-frequency images [2509.19719]. The paper follows XNet-style wavelet decomposition and states that the transform type is DWT rather than FFT or DCT [2509.19719]. A standard single-level separable 2D DWT formulation is provided in the source to clarify the process:

- $S_{LL}[m,n] = \sum_{x=0}^{M-1} \sum_{y=0}^{N-1} f(x,y)\, h_L[2m-x]\, h_L[2n-y]$
- $S_{LH}[m,n] = \sum_{x=0}^{M-1} \sum_{y=0}^{N-1} f(x,y)\, h_L[2m-x]\, h_H[2n-y]$
- $S_{HL}[m,n] = \sum_{x=0}^{M-1} \sum_{y=0}^{N-1} f(x,y)\, h_H[2m-x]\, h_L[2n-y]$
- $S_{HH}[m,n] = \sum_{x=0}^{M-1} \sum_{y=0}^{N-1} f(x,y)\, h_H[2m-x]\, h_H[2n-y]$

In this formulation, the low-frequency image is defined as $I_L = S_{LL}$, while the high-frequency image $I_H$ is constructed from the high-frequency subbands; the exact stacking scheme is not specified in the paper, which states only that it follows XNet-style HF/LF decomposition [2509.19719]. That omission matters for reproducibility, and the paper itself lists the unspecified wavelet type and decomposition level as a limitation [2509.19719].

The visual encoder is dual-branch. A ConvNeXt-Tiny backbone processes the LF image and another ConvNeXt-Tiny backbone processes the HF image [2509.19719]. Both branches produce multi-scale features at downsampling rates 4, 8, 16, and 32:
$F_m^1 \in \mathbb{R}^{H/4 \times W/4 \times C_1}$,
$F_m^2 \in \mathbb{R}^{H/8 \times W/8 \times C_2}$,
$F_m^3 \in \mathbb{R}^{H/16 \times W/16 \times C_3}$,
$F_m^4 \in \mathbb{R}^{H/32 \times W/32 \times C_4}$,
for $m \in \{LF, HF\}$ [2509.19719].

The representational division is explicit. Low-frequency components emphasize “large-scale structure and context,” whereas high-frequency components preserve “fine textural detail and boundaries” [2509.19719]. In the intended medical setting, this complementarity is directly tied to segmentation quality: LF supports coarse lesion localization and organ-level context, while HF supports lesion edge disambiguation and the recovery of subtle structures [2509.19719].

## 3. Visual-frequency interaction through FFBI

The Frequency-domain Feature Bidirectional Interaction (FFBI) module is the visual fusion component of FMISeg [2509.19719]. It operates at the last encoder stage and exchanges information between the deepest LF and HF feature maps via bidirectional multi-head cross-attention [2509.19719].

The inputs are:
- $F_{LF}^4 \in \mathbb{R}^{H/32 \times W/32 \times C_4}$
- $F_{HF}^4 \in \mathbb{R}^{H/32 \times W/32 \times C_4}$

The paper gives the FFBI equations as:
- $\hat{F}_{HF}^4 = LN(F_{HF}^4 + MHCA(F_{HF}^4, F_{LF}^4, F_{LF}^4))$
- $\hat{F}_{LF}^4 = LN(F_{LF}^4 + MHCA(F_{LF}^4, F_{HF}^4, F_{HF}^4))$ [2509.19719]

The multi-head cross-attention operator is described using standard scaled dot-product attention. For head $i$,
- $Q_i = F_Q W_Q^{(i)}$
- $K_i = F_K W_K^{(i)}$
- $V_i = F_V W_V^{(i)}$
- $Attention_i = softmax(Q_i K_i^\top / \sqrt{d_h}) V_i$
- $MHCA(Q,K,V) = Concat_i(Attention_i) W_O$ [2509.19719]

The purpose of FFBI is asymmetrical but bidirectional. HF receives semantic and global guidance from LF, which helps “reject HF noise and sharpen boundaries in context,” while LF receives fine-detail cues from HF, improving localization and border accuracy [2509.19719]. Residual connections and layer normalization preserve the original signals and stabilize fusion [2509.19719].

The ablation results reported for FMISeg support the claim that HF and LF are complementary. On QaTa-COV19 and MosMedData+, the two-branch model with FFBI outperforms single-branch raw-image, HF-only, LF-only, and simple two-branch concatenation baselines [2509.19719]. The source reports the following Dice/mIoU values:

| Configuration | QaTa-COV19 | MosMedData+ |
|---|---:|---:|
| Single-branch raw image | 89.86 / 81.72 | 78.21 / 64.17 |
| Single-branch HF only | 88.75 / 80.15 | 77.16 / 63.04 |
| Single-branch LF only | 89.54 / 81.23 | 77.89 / 63.69 |
| Two-branch concat(HF, LF) | 90.61 / 82.93 | 78.65 / 64.88 |
| Two-branch with FFBI | 91.21 / 83.84 | 79.30 / 65.71 |

These results are used in the paper to conclude that “HF/LF are complementary” and that FFBI’s bidirectional cross-attention is “superior to simple concatenation” [2509.19719].

## 4. Language-guided decoding through LFFI

FMISeg’s textual pathway uses CXR-BERT to encode clinical text into word-level features $F_T \in \mathbb{R}^{L \times C}$ [2509.19719]. The paper specifies that word-level features are used and that CLS pooling is not used [2509.19719]. This textual representation is injected late, in the decoder, through the Language-guided Frequency-domain Feature Interaction (LFFI) module [2509.19719].

At a decoder stage, LFFI receives a visual feature map $F_m \in \mathbb{R}^{h \times w \times C}$ for $m \in \{LF, HF\}$ and the word-level textual features $F_T \in \mathbb{R}^{L \times C}$ [2509.19719]. It first performs bidirectional cross-attention:
- $F_m' = MHCA(F_m, F_T, F_T)$
- $F_T' = MHCA(F_T, F_m, F_m)$ [2509.19719]

It then computes an interaction tensor and a learned gate:
- $F_M = F_m' \otimes (F_T')^\top$
- $W = \delta(Linear(F_M))$
- $F_m^o = Conv(F_m + F_m' \odot W)$ [2509.19719]

The paper interprets this gating mechanism as a “semantically irrelevant filter” that suppresses text-irrelevant visual content [2509.19719]. Cross-attention alone may import noise and lacks positional granularity; the gate therefore uses the interaction between visual and textual attended features to generate spatially varying, channel-wise reweighting [2509.19719].

The design is explicitly late fusion. Frequency fusion occurs at the deepest encoder stage through FFBI, while text fusion occurs progressively through LFFI in the decoder [2509.19719]. The paper reports that adding more LFFI layers from high to low decoder levels yields incremental performance improvements [2509.19719]. The corresponding ablation on QaTa-COV19 and MosMedData+ shows a monotonic progression from “No text” to “4 layers”:

| LFFI depth | QaTa-COV19 | MosMedData+ |
|---|---:|---:|
| No text | 87.63 / 78.13 | 76.45 / 62.87 |
| 1 LFFI layer | 90.64 / 82.98 | 78.71 / 64.93 |
| 2 layers | 90.86 / 83.26 | 78.97 / 65.25 |
| 3 layers | 91.06 / 83.63 | 79.16 / 65.48 |
| 4 layers | 91.21 / 83.84 | 79.30 / 65.71 |

The paper concludes that “progressive late fusion of text improves segmentation” and that “more LFFI layers yield incremental gains” [2509.19719].

This decoder-centric language integration parallels a broader design choice in multimodal segmentation. In UFAFormer, a unified decoder with symmetric cross-modal interaction modules is used to align visual and textual forgery cues [2309.09667]. In FAVS, cross-modal consistency is enforced progressively after frequency-domain decomposition and recomposition [2509.18912]. FMISeg differs in its specific domain and architectural details, but the common principle is that language or another complementary modality should modulate frequency-enriched visual features after sufficient visual abstraction has been established.

## 5. Training setup, datasets, and empirical performance

FMISeg is evaluated on two datasets. QaTa-COV19 contains “9,258 CXR images with lesion annotations and textual descriptions,” split into 5,716 training, 1,429 validation, and 2,113 test images [2509.19719]. MosMedData+ contains “2,729 COVID-19 CT slices with binary masks,” split into 2,183 training, 273 validation, and 273 test images following prior language-guided medical image segmentation work [2509.19719].

The training setup reported in the paper is precise. Input resolution is $224 \times 224$ for fair comparison [2509.19719]. The framework is implemented in PyTorch, uses ConvNeXt-Tiny for both LF and HF branches, AdamW optimization, an initial learning rate of $3\times10^{-4}$ with cosine annealing down to $1\times10^{-6}$, batch size 32, hidden dimension 768 in the interaction modules, and NVIDIA RTX 3090 hardware [2509.19719]. The paper does not detail intensity normalization or augmentation policies [2509.19719].

The objective function combines Dice loss and binary cross-entropy [2509.19719]. The source gives:
- $L_{Dice} = 1 - \frac{2 \sum_i p_i g_i + \epsilon}{\sum_i p_i + \sum_i g_i + \epsilon}$
- $L_{BCE} = - \frac{1}{N} \sum_{i=1}^N [g_i \log p_i + (1 - g_i)\log(1 - p_i)]$ [2509.19719]

No auxiliary frequency-regularization or cross-modal alignment losses are introduced in FMISeg itself [2509.19719]. That is notable because related work such as FOCA includes a composite objective with mask, classification, text-generation, and contrastive terms [2602.18880], while the FMISeg formulation remains limited to dense prediction supervision.

The main quantitative comparison reported by the paper is as follows:

| Method | QaTa-COV19 Dice / mIoU | MosMedData+ Dice / mIoU |
|---|---:|---:|
| FMISeg | 91.21 / 83.84 | 79.30 / 65.71 |
| nnUNet | 80.42 / 70.81 | 72.59 / 60.36 |
| LGA | 84.65 / 76.23 | — |
| TGCAM | 90.60 / 82.81 | 77.82 / 63.69 |

The paper states that FMISeg achieves gains of “+10.79 Dice, +13.03 mIoU” over nnUNet on QaTa-COV19 and “+6.71 Dice, +5.35 mIoU” on MosMedData+ [2509.19719]. Against TGCAM, the reported improvements are “+0.61 Dice, +1.03 mIoU” on QaTa-COV19 and “+1.48 Dice, +2.02 mIoU” on MosMedData+ [2509.19719]. The qualitative comparison is summarized by the claim that FMISeg produces “cleaner boundaries, fewer missed small lesions, and better suppression of false positives in complex regions” than uni-modal and language-guided baselines [2509.19719].

## 6. Relation to adjacent frequency-aware multimodal models

Although FMISeg is a medical image segmentation model, its design sits within a larger family of frequency-aware multimodal architectures that differ in modality pairings, task definitions, and fusion operators.

UFAFormer addresses “Detecting and Grounding Multi-modal Manipulation” by introducing a discrete wavelet transform-based frequency encoder, a forgery-aware mutual module, and a unified decoder for image-text reasoning [2309.09667]. Its detailed architecture includes intra-band and inter-band self-attentions over DWT sub-bands, mutual cross-attention between RGB and frequency features, and symmetric text-visual cross-modal interaction modules [2309.09667]. The paper’s adaptation notes explicitly state that these design choices “directly inform a Frequency-domain Multi-modal Interaction Segmentation model (FMISeg),” namely by retaining the DWT frequency encoder and mutual alignment machinery while replacing detection heads with a segmentation head [2309.09667]. This is not the same FMISeg as the medical segmentation model of 2025, but it establishes a second usage of the term as an adaptation blueprint for dense localization of manipulated regions.

FAVS reformulates audio-visual segmentation as a “frequency-domain decomposition and recomposition problem” [2509.18912]. Its Frequency-Domain Enhanced Decomposer separates features into high-, mid-, low-frequency bands and a residual via residual-based iterative decomposition, while the Synergistic Cross-Modal Consistency module uses a mixture-of-experts architecture with dynamic expert routing to align audio and visual features [2509.18912]. The model reports state-of-the-art performance on AVSBench subsets and attributes gains to better handling of the modality-specific contradiction that “audio high-frequency signals” are often noisy while “visual high-frequency signals” are structurally informative [2509.18912]. Relative to FMISeg, this framework broadens the scope of frequency-aware multimodal interaction from language-guided medical imaging to audio-visual dense prediction.

FOCA extends frequency-aware multimodality into detection, localization, and explanation using a multimodal large language model [2602.18880]. It uses DWT to extract frequency cues, with $x_{HH}$ serving as the query in a Frequency Attention Fusion cross-attention:
$$
A=\operatorname{softmax}\left(\frac{(\mathbf{w}_q x_{HH})(\mathbf{w}_k x_{img})^\top}{\sqrt{d_k}}\right)\mathbf{w}_v x_{img}
$$
and residual fusion
$$
x_f=\operatorname{proj}(A)+x_{img}
$$
[2602.18880]. FOCA’s use of explicit segmentation tokens, a frozen vision encoder, and joint mask-generation and explanation objectives suggests a different trajectory for frequency-aware multimodal interaction: coupling spectral guidance with instruction-conditioned reasoning and interpretability [2602.18880].

Taken together, these systems show three recurring principles. First, frequency-domain representations are used to preserve information that may be weakly expressed in the spatial domain. Second, multimodal fusion is usually not simple concatenation; it is mediated by cross-attention, mutual alignment, or expert routing. Third, dense prediction quality is improved when frequency information remains spatially aligned, as in DWT-based formulations [2309.09667] [2509.18912] [2509.19719] [2602.18880]. This suggests that FMISeg is best understood not as an isolated architecture but as one instance of a broader methodological pattern.

## 7. Limitations, misconceptions, and plausible directions

A common misconception is that FMISeg is merely a conventional language-guided segmentation model with an added wavelet preprocessing step. The reported architecture contradicts that interpretation. The frequency decomposition is structurally embedded into the model through a dual-branch encoder, FFBI bidirectional interaction, and LFFI-based language-guided filtering in the decoder [2509.19719]. The performance gap between simple HF/LF concatenation and FFBI also indicates that the claimed benefit is not attributed to decomposition alone but to the interaction mechanism built on top of it [2509.19719].

Another possible misconception is that “frequency-domain” in FMISeg denotes a generic spectral transform. The paper explicitly notes that FMISeg does not use FFT or DCT and instead adopts DWT to form LF and HF images [2509.19719]. The reason, stated indirectly through the architectural design, is that DWT preserves spatial structure in a way that can be exploited by encoder-decoder segmentation pipelines. This is consistent with UFAFormer’s use of DWT to preserve spatial layout while exposing texture and edge anomalies [2309.09667].

The limitations reported for FMISeg are concrete. The wavelet type and decomposition level are “not explicitly stated,” which affects reproducibility [2509.19719]. The experiments focus on “2D slices (CXR, CT slices),” so extension to 3D volumetric segmentation would require “3D wavelet transforms and 3D attention, with memory considerations” [2509.19719]. Only English chest-report domain text via CXR-BERT is used, and sensitivity to report quality or prompt formulation is not analyzed [2509.19719]. The paper also notes that robustness to noisy or incomplete reports is not formally tested [2509.19719].

Several future directions are identified in the surrounding literature rather than established as facts for FMISeg itself. The medical FMISeg paper points to broader anatomy, multi-disease settings, and richer clinical narratives as natural extensions [2509.19719]. The FAVS work discusses adaptive band selection, learnable filter banks, multi-scale frequency fusion, and explicit frequency-domain alignment losses as possible strengthenings of a general FMISeg design [2509.18912]. FOCA, meanwhile, suggests that frequency-aware segmentation can be coupled to multimodal large language models for explanation and cross-domain interpretability [2602.18880]. These are not part of FMISeg as reported, but they indicate plausible directions in which the architecture class may evolve.

In that sense, FMISeg occupies a specific and technically well-defined point in the development of frequency-aware multimodal segmentation: it operationalizes wavelet-domain decomposition, bidirectional LF/HF interaction, and decoder-stage language guidance for pulmonary infection lesion segmentation, while leaving open questions about transform specification, volumetric generalization, textual robustness, and more explicit cross-modal alignment objectives [2509.19719].

Source: https://www.emergentmind.com/topics/frequency-domain-multi-modal-interaction-model-fmiseg