Spectral Adaptive Idempotent Decoupling (SAID)

• SAID is a spectral-domain module that disentangles domain-invariant semantic content from modality-specific style using frequency masking and idempotent filter design.
• It employs hard, soft, and free filtering modes—leveraging FFT, Gaussian filter banks, and U-Net architectures—to adaptively partition feature maps.
• Self-distillation-driven decoupling loss and integration with spatial-spectral frameworks enhance unsupervised domain adaptive object detection, boosting mAP by up to +3.91%.

Spectral Adaptive Idempotent Decoupling (SAID) is a spectral-domain module for decoupling domain-invariant (DI) and domain-specific (DS) features in unsupervised domain adaptive object detection (UDAOD) across visible and infrared (RGB–IR) modalities. SAID employs frequency-based masking, idempotent filter design, and self-distillation-driven regularization to yield interpretable and adaptive feature decomposition, enabling robust transfer between domains characterized by substantial spectral and style discrepancies (Zhang et al., 16 Jul 2025).

1. Motivation and Theoretical Foundations

In cross-modal domain adaptation, particularly for RGB–IR object detection, differences between domains stem largely from photometric and sensor-induced “style” shifts (illumination, noise, spectral signatures) that localize in the high-frequency components of the feature spectrum, while “content” or semantic structure (object boundaries, layout) resides predominantly in low-frequency components. SAID seeks to explicitly disentangle these factors by projecting feature maps into frequency space and partitioning them into DI and DS subspaces: $H_{\textrm{inv}}$ for modality-invariant semantic content, and $H_{\textrm{spe}}$ for style/noise.

This interpretable partition is motivated by the observation that aligning only global feature distributions disregards heterogeneous subdomain shifts (e.g., daytime/nighttime variation), leading to suboptimal adaptation. Spectral decoupling allows for stable modality-agnostic representation extraction through explicit frequency-band separation (Zhang et al., 16 Jul 2025).

2. Mathematical Formulation

Let $X \in \mathbb{R}^{C \times H \times W}$ denote a convolutional feature map. SAID leverages 2D FFT ($A = \text{FFT2}(X)$) to obtain the complex spectrum $$A(u, v) = \text{Amp}(u, v) \cdot e^{j \cdot \text{Phase}(u, v)}$$. The objective is to construct masks $H_{\textrm{inv}}(u, v)$ and $$H_{\textrm{spe}}(u, v) = 1 - H_{\textrm{inv}}(u, v)$$, satisfying the idempotence property $H \odot H = H$. Three operational modes are provided:

• Hard mode: Gaussian low-pass and high-pass filters are instantiated as fixed masks, e.g., $G_L(u, v) = e^{-D^2/2\sigma_l^2}$ for low frequencies, $G_H(u, v) = 1 - e^{-D^2/2\sigma_h^2}$ for high frequencies, where $H_{\textrm{spe}}$0 indicates normalized spectral distance.
• Soft mode: A learnable filter bank partitions the radial frequency range into $H_{\textrm{spe}}$1 overlapping Gaussian bands; a neural module regresses weights $H_{\textrm{spe}}$2 (via Conv→FC layers) to form the adaptive low-frequency mask: $H_{\textrm{spe}}$3, with $H_{\textrm{spe}}$4 defined as a Gaussian difference.
• Free mode: A lightweight U-Net directly generates $H_{\textrm{spe}}$5 from concatenated Amp and Phase channels.

Filtering is performed by masking $H_{\textrm{spe}}$6 to yield the separated spectra: \begin{align*} \text{DI}\textrm{spectrum} &= A \odot H{\textrm{inv}} \ \text{DS}\textrm{spectrum} &= A \odot H{\textrm{spe}} \end{align*} which are returned to feature space via the inverse FFT.

3. Filter Bank-Based Spectral Processing Paradigm

The SAID filter-bank paradigm implements the following workflow:

1. Compute the FFT: $H_{\textrm{spe}}$7.
2. Extract amplitude and phase, concatenate, and process through a convolutional flow to predict the filter weights $H_{\textrm{spe}}$8.
3. Precompute the Gaussian filter bank $H_{\textrm{spe}}$9 for the image size and $X \in \mathbb{R}^{C \times H \times W}$0 bands.
4. Synthesize $X \in \mathbb{R}^{C \times H \times W}$1 and $X \in \mathbb{R}^{C \times H \times W}$2.
5. Apply filters to obtain DI and DS spectra; invert via IFFT to $X \in \mathbb{R}^{C \times H \times W}$3 and $X \in \mathbb{R}^{C \times H \times W}$4 in feature space.
6. Backpropagate detection and decoupling gradients to train the filter regressor (soft mode).

This modular design supports both adaptive and pre-fixed filter strategies and promotes differentiable, interpretable spectral disentanglement (Zhang et al., 16 Jul 2025).

4. Self-Distillation-Driven Decoupling Loss

SAID incorporates an idempotent self-distillation objective to enforce strict separation and consistency of DI and DS features. This is operationalized by repeated (three-pass) application of the same SAID module (weight-tied) to the DI feature, generating $X \in \mathbb{R}^{C \times H \times W}$5, and $X \in \mathbb{R}^{C \times H \times W}$6 (and likewise for DS). The core decoupling loss uses Pearson correlation coefficients (PCC) to maximize DI feature consistency and minimize DS feature correlation across passes: $X \in \mathbb{R}^{C \times H \times W}$7 where $X \in \mathbb{R}^{C \times H \times W}$8 controls the relative strength and $X \in \mathbb{R}^{C \times H \times W}$9 avoids division by zero.

The burn-in stage loss combines detection and decoupling terms: $A = \text{FFT2}(X)$0. In mean-teacher mutual learning, the $A = \text{FFT2}(X)$1 term is also applied to both source and target samples.

5. Integration with Spatial-Spectral Decoupling and Coupling (SS-DC) Framework

Within the SS-DC framework for UDAOD, SAID forms the spectral branch, which operates in parallel with a spatial backbone. Key integration mechanisms include:

• Spatial-Spectral Coupling: At low levels, DI-based attention is computed from $A = \text{FFT2}(X)$2 and fused with spatial features via a convex combination. At high levels, cross-attention between DI features and spatial backbone features is employed, updating the outputs $A = \text{FFT2}(X)$3 at each level $A = \text{FFT2}(X)$4.
• DS Guidance Tokens: The DS features are embedded to generate guidance tokens $A = \text{FFT2}(X)$5 that are prepended to the DETR encoder input. This mechanism guides the transformer to suppress modality-specific noise or interference during detection.

A summary table of modes and their optimal settings, as found on FLIR-ADAS (Zhang et al., 16 Jul 2025):

Mode	mAP (%)	N Bands
Hard (fixed)	46.31	N/A
Soft (bank)	48.02	100
Free (U-Net)	47.15	N/A

6. Experimental Results and Comparison

Experiments on FLIR-ADAS and FLIR datasets demonstrate the quantitative impact of SAID and the full SS-DC framework. On FLIR-ADAS, integrating SAID yields a +3.91 mAP improvement over the Semi-DETR baseline (45.16% vs. 41.25%). Full SS-DC achieves 48.02% mAP, exceeding previous methods such as D3T (43.04%). On FLIR, SS-DC obtains 72.43% mAP versus 69.30% for D3T.

Further ablations confirm the superiority of soft-mode filtering (filter-bank, 48.02% mAP) over hard (46.31%) or U-Net (47.15%) modes, and identify $A = \text{FFT2}(X)$6 filter bands as optimal; increasing $A = \text{FFT2}(X)$7 beyond this degrades performance. The optimal self-distillation hyperparameters are $A = \text{FFT2}(X)$8, $A = \text{FFT2}(X)$9. These results suggest that both adaptive spectral decomposition and self-distillation regularization are critical for robust, interpretable RGB–IR domain adaptation.

7. Significance and Applicability

SAID establishes a spectral, adaptive, and idempotent paradigm for cross-modal feature disentanglement in deep networks, advancing interpretable UDAOD by operationalizing classic image processing intuition in a learnable end-to-end module. The demonstrated ability to separate domain-invariant semantic structure from modality-induced style provides a principled foundation for future domain adaptation systems involving other multi-modal or challenging spectral distribution settings. Empirical findings show that careful spectral decoupling, idempotent filter design, and self-distillation yield state-of-the-art results on domain adaptation benchmarks (Zhang et al., 16 Jul 2025).