Hierarchical Frequency Sampling Module

• Hierarchical Frequency Sampling Module is a learnable component that decomposes, filters, and integrates frequency bands at multiple scales to enhance semantic detail and context.
• It employs a pipeline of Fourier transforms, adaptive filtering, and selective band sampling to preserve high-frequency details and balance information across network layers.
• Empirical results show improvements in detection mAP and segmentation mean IoU by efficiently maintaining high-frequency boundary details and global context.

A Hierarchical Frequency Sampling Module (HFS) is a learnable architectural component that manages the selective decomposition, adaptive manipulation, and multi-scale integration of frequency information within neural networks. HFS schemes are designed to address challenges in extracting, preserving, and combining features across spectral bands and spatial hierarchy, notably in vision tasks requiring precise boundary semantics, long-range consistency, or robust generalization. Variants have been deployed in object detection, semantic segmentation, face anti-spoofing, and surrogate modeling for oscillatory dynamical systems, emphasizing scale-aware content preservation and suppression of undesirable spectral components.

1. Theoretical Foundations and Motivation

Traditional deep learning architectures face limitations in maintaining feature consistency at multiple semantic levels, resulting in semantic gaps and suboptimal performance at object boundaries or in domains with frequency-sensitive phenomena. HFS modules address these issues by explicitly decomposing features into frequency bands and adaptively sampling or filtering according to the information content and semantic requirements of each stage.

For example, in object detection, shallow feature maps capture fine detail but are prone to noise, while deeper layers encapsulate semantic context but may lose high-frequency boundary information. HFS ensures that high-frequency bands are preserved in early layers, mid-band information is emphasized in intermediate stages, and low-frequency content is prioritized in deeper levels. This hierarchical management enforces cross-scale frequency consistency and enhances both local and global feature utility (Lin et al., 12 Jul 2025).

In dynamical systems modeling, a two-layer HFS design enables balanced sampling across regions with oscillatory and non-oscillatory behaviors, enriching boundary data and capturing sensitive transitions in frequency prediction tasks (Rao et al., 2024). Semantic segmentation approaches use HFS mechanisms to prevent aliasing of high-frequency components during down/upsampling, thereby preserving region boundaries and textual detail (Chen et al., 16 Jul 2025).

2. Mathematical Formulation and Algorithmic Steps

A canonical HFS pipeline operates as follows:

1. Frequency Decomposition: For each feature map $X_k\in\mathbb{R}^{C\times H_k\times W_k}$, a 2D Discrete Fourier or Cosine Transform is applied:

$$F_k(u,v) = \mathcal{F}\bigl[X_k(h,w)\bigr]_{(u,v)} = \sum_{h=0}^{H_k-1}\sum_{w=0}^{W_k-1}X_k(h,w)\,e^{-2\pi j\,(u\,h/H_k + v\,w/W_k)}$$

This yields a frequency-domain representation indexed by $(u,v)$.

2. Adaptive Frequency Filtering or Modulation: For adaptive refinement, learned complex filters $H_k(u,v)$ are applied:

$\widetilde F_k(u,v) = H_k(u,v) \cdot F_k(u,v)$

with $H_k(u,v) = L_k(u,v) + A_k(u,v)$, where $L_k$ dampens low frequencies and $A_k$ amplifies high frequencies. These filters are parameterized by shallow convolutions over the spectral grid (Lin et al., 12 Jul 2025).

3. Frequency Band Sampling: Select an index set $\Omega_k$ corresponding to the desired frequency band for pyramid level $k$. Define a sampling operator $S_k$:

$$S_k(\widetilde F_k) = \{\widetilde F_k(u,v)\ |\ (u,v)\in\Omega_k\}$$

Frequencies outside $\Omega_k$ are zeroed. For example, shallow layers select high-frequency bands, deep layers select low-frequency bands, and intermediate layers emphasize mid-bands. Typical cut-offs are set as fractions (e.g., 0.1, 0.4) of the Nyquist radius (Lin et al., 12 Jul 2025).

4. Inverse Transform and Reconstruction: Reconstruct the spatial map:

$$\widetilde X_k(h,w) = \mathcal{F}^{-1}[\widehat F_k(u,v)] = \frac{1}{H_k W_k}\sum_{u=0}^{H_k-1}\sum_{v=0}^{W_k-1}\widehat F_k(u,v)\,e^{2\pi j(u h/H_k + v w/W_k)}$$

$\widehat F_k$ indicates frequency coefficients post sampling.

5. Hierarchical Fusion: Filtered features from each level ($\{\widetilde X_2, \ldots, \widetilde X_5\}$) are fused in subsequent modules (e.g., progressive hierarchical fusion networks) to integrate multi-scale context (Lin et al., 12 Jul 2025).

A representative implementation is as follows:

Input: X_k F_k = DFT(X_k) H_k = CLFD_trigger(F_k) + CHFA_trigger(F_k) F_tilde_k = H_k * F_k F_hat_k = S_k(F_tilde_k) X_tilde_k = IDFT(F_hat_k) Output: X_tilde_k

In frequency-aware sampling for surrogate modeling of oscillatory systems, a two-layer approach organizes the candidate set by local sensitivity (“gradient degree”), then iteratively refines high-residual regions via genetic search over the coefficient space (Rao et al., 2024).

3. Hierarchical Band Selection and Adaptation

The core principle of HFS is the assignment of specific frequency bands to different semantic levels in a hierarchical network:

Pyramid Level	$\Omega_k$ Selection	Semantic Role
Shallow (e.g., P2)	$\sqrt{u^2/H_2^2 + v^2/W_2^2} \geq \tau_{high}$	Preserve high-frequency detail
Mid (e.g., P3–P4)	$$\tau_{low} \leq \sqrt{u^2/H_k^2 + v^2/W_k^2} \leq \tau_{high}$$	Emphasize mid-band content
Deep (e.g., P5)	$\sqrt{u^2/H_5^2 + v^2/W_5^2} \leq \tau_{low}$	Focus on low-frequency semantic content

Typical $\tau_{low}, \tau_{high}$ values are 0.1 and 0.4 of the Nyquist frequency. This structure allows the module to enforce frequency consistency and mitigate the semantic gap between scales (Lin et al., 12 Jul 2025). In vision transformers and segmentation networks, analogous sampling/remapping is achieved using attention-guided non-uniform grids, driven by high-frequency saliency maps to adaptively protect detail prior to downsampling (Chen et al., 16 Jul 2025).

4. Integration Strategies and Implementation

HFS modules are typically not isolated backbone or head components but are embedded within neck blocks of the network or wrap around critical encoder-decoder transitions. Key implementation points include:

• Positioning: In Butter, HFS is inside each FAFCE block in the neck. Post-backbone, each feature map passes through HFS before hierarchical fusion (PHFFNet) (Lin et al., 12 Jul 2025).
• Computation: Global 2D FFT/DFT is computed across each map. CLFD and CHFA triggers are realized as $3\times3$ convolutions that produce spatially varying frequency weights. Each $\Omega_k$ typically retains 20–50% of frequencies; indices are contiguous spectral rings.
• Efficiency: Each FAFCE block with HFS adds $\sim$1.5 GFLOPs and 5.4 M parameters, with the full Butter neck costing 31 GFLOPs (Lin et al., 12 Jul 2025).
• In segmentation, ARS modules precede every stride-2 (downsampling) layer, and MSAU modules demodulate in the upsampling path. Insertions are backbone- and resolution-agnostic (Chen et al., 16 Jul 2025).

5. Applications and Empirical Effects

Object Detection

In Butter (Lin et al., 12 Jul 2025), HFS improves detection mAP@50 by +1.2 on KITTI and +1.6 on Cityscapes over prior state-of-the-art, while reducing parameter count by 64% (vs. Hyper-YOLO-S on Cityscapes). Ablation studies isolating HFS show that its removal degrades both accuracy and feature consistency.

Semantic Segmentation

Integration of hierarchical frequency sampling as spatial frequency modulation (SFM) yields consistent improvements in mean IoU (+1.1–5.0 on Cityscapes) and boundary metrics (BF-score, Boundary-IoU) across various architectures. Visualization confirms crisper edges and more consistent regions post-processing. Classification and instance/panoptic segmentation tasks also benefit from detail preservation and reduced aliasing artifacts (Chen et al., 16 Jul 2025).

Surrogate Modelling of Oscillatory Systems

A two-layer HFS design consisting of gradient-based filtering and multi-grid genetic refinement produces a compact, balanced training set, improving RMSE by 13–71% and drastically reducing class imbalance and label diversity (Imbalance Ratio from 11.8→2.9, Gini Index from 0.94→0.77 in the Activator-Inhibitor system). Boundary error is reduced by up to 66% over competing sampling methods (Rao et al., 2024).

6. Limitations and Prospective Directions

Known constraints of current HFS approaches include computational overhead from repeated FFT/IFFT operations and static frequency-band selection that may not always reflect content adaptivity. Proposed future work includes the migration to spatial-domain learned filterbanks (e.g., separable DCT) for reduced inference latency and the development of dynamic, content-aware $\Omega_k$ selectors. Extension to spatio-temporal frequency sampling for video detection remains an open avenue (Lin et al., 12 Jul 2025).

A plausible implication is that hierarchical frequency sampling frameworks will find increasing utility as model demands for high semantic fidelity, boundary preservation, and efficient multi-scale integration intensify across vision, signal processing, and scientific modeling.

7. Comparative Summary of Module Variants

Module	Domain	Sampling/Decomposition	Hierarchical Aspect	Empirical Effect	Ref.
HFS (FAFCE in Butter)	Object Detection	FFT, adaptive filter, ring bands	FPN-level, frequency bands	+1.2–1.6 mAP@50, 64% fewer params	(Lin et al., 12 Jul 2025)
HGGS (2-layer)	Neural Surrogates	Gradient, GMM, genetic	Sensitivity-driven, boundaries	-13–71% RMSE, large imbalance reduction	(Rao et al., 2024)
SFM (ARS+MSAU)	Segmentation, CLS	Non-uniform grid, attention-guided	Pre/post stride, multi-scale	+1.1–5.0 mIoU (seg.), crisper boundaries	(Chen et al., 16 Jul 2025)

This diversity underscores the extensibility of HFS as an architectural paradigm, with unified emphasis on frequency-adaptive, hierarchical information management.