Y-Shaped Frequency Prediction Network (Y-FPN)

• The paper introduces Y-FPN as a lightweight module that predicts pixel-wise low- and high-frequency priors to guide reverse diffusion in dense prediction tasks.
• Its Y-shaped architecture features a shared encoder branching into dual heads to separately extract coarse semantic information and fine details.
• Trained with FrequencyMix, Y-FPN enhances continual test-time adaptation by conditioning denoising to preserve task-relevant semantic structure and mitigate catastrophic forgetting.

Y-Shaped Frequency Prediction Network (Y-FPN) is the lightweight learned module at the core of FOCUS, a frequency-based conditioning approach within a diffusion-driven input-adaptation framework for continual test-time adaptation (CTTA). Its function is to take a corrupted or noisy image and predict pixel-wise frequency priors that indicate which image content should be treated as low-frequency content and which should be treated as high-frequency content during reverse denoising. In FOCUS, these learned priors condition the reverse diffusion process so that denoising preserves task-relevant semantic information for dense prediction rather than washing it away (Tjio et al., 20 Aug 2025).

1. Position within continual test-time adaptation

FOCUS is an input adaptation method for CTTA. Rather than updating the task model directly on noisy test images, it first uses a pretrained diffusion model to denoise the input into a more source-like image and then feeds that denoised image to the task model. Within this pipeline, Y-FPN supplies the learned, spatially adaptive frequency priors that govern how denoising should preserve semantic structure (Tjio et al., 20 Aug 2025).

The motivation is specific to dense prediction, especially semantic segmentation and monocular depth estimation. The paper argues that the semantics required for these tasks are encoded in both low-frequency information and high-frequency information. Low-frequency information corresponds to coarse color, illumination, and region structure, whereas high-frequency information corresponds to edges, boundaries, and fine object details. This departs from classification-oriented denoising settings in which low-frequency preservation can be sufficient. For dense prediction, pixel-level conditioning is required because object boundaries, local texture, spatial consistency, and coarse scene layout all affect downstream predictions.

This framing also clarifies why Y-FPN is central to catastrophic forgetting mitigation. FOCUS itself is not a model adaptation method; it does not primarily rely on continual finetuning of the task model. A plausible implication is that its robustness derives from altering the input distribution seen by the predictor rather than repeatedly modifying the predictor parameters. The paper explicitly links this design to reduced forgetting risk in continual settings (Tjio et al., 20 Aug 2025).

2. Architecture and outputs

Y-FPN is called “Y-shaped” because it consists of a shared stem or encoder that processes the corrupted image and then branches into two heads: one branch predicts low-frequency priors and the other predicts high-frequency priors. The network is described as sharing a similar architecture with UNet, but with two separate output branches producing two per-pixel kernels, $\mathbf{K}_{i=1}$ for low-frequency priors and $\mathbf{K}_{i=2}$ for high-frequency priors. It is lightweight, with 27.5M parameters (Tjio et al., 20 Aug 2025).

Its input is a corrupted image

$\mathbf{I}^c \in \mathbb{R}^{H \times W \times 3}.$

For every pixel, Y-FPN predicts two spatially adaptive kernels of size $K \times K$, with $K=3$:

$$\mathbf{K}_{i=1}, \mathbf{K}_{i=2} \in \mathbb{R}^{H \times W \times K^2}.$$

These kernels are converted into the low-frequency prior $\Phi_{\text{lp}}$ and the high-frequency prior $\Phi_{\text{hp}}$. The paper also describes the output priors figure-wise as $K \times K \times 3$ per pixel for an RGB image.

Component	Specification	Role
Input	$\mathbf{I}^c \in \mathbb{R}^{H \times W \times 3}$	Corrupted/noisy RGB image
Low-frequency branch	$\mathbf{K}_{i=2}$0, $\mathbf{K}_{i=2}$1	Predicts adaptive kernels for $\mathbf{K}_{i=2}$2
High-frequency branch	$\mathbf{K}_{i=2}$3, $\mathbf{K}_{i=2}$4	Predicts adaptive kernels for $\mathbf{K}_{i=2}$5

The architectural choice reflects a kernel prediction network formulation rather than a fixed filtering prior. This suggests that the representation of “frequency” in Y-FPN is not global and handcrafted, but local, learned, and spatially varying across the image.

3. Frequency prior construction

Y-FPN generates per-pixel filtering kernels directly from the noisy input:

$\mathbf{K}_{i=2}$6

To regularize the predicted kernels, each kernel is normalized with a spatial softmax:

$\mathbf{K}_{i=2}$7

The low-frequency prior is then computed by applying the first kernel as an adaptive local smoother:

$\mathbf{K}_{i=2}$8

The high-frequency prior is computed as a residual-like signal:

$\mathbf{K}_{i=2}$9

The interpretation given in the paper is that the low-frequency branch learns how to preserve smooth, large-scale semantic content, while the high-frequency branch captures fine details, edges, and boundary structure by preserving complementary residual information (Tjio et al., 20 Aug 2025). The low-frequency prior is generally sufficient for preserving semantics, but adding the high-frequency prior can help on corruptions that damage edges or contrast.

This decomposition is significant because it is spatially adaptive. Unlike a predefined low-pass operator, Y-FPN predicts per-pixel kernels from the observed corruption itself. A plausible implication is that the same image region may be treated differently depending on local corruption structure, which is particularly relevant for dense prediction under heterogeneous degradations.

4. Conditioning the reverse diffusion process

FOCUS uses a pretrained diffusion model to iteratively denoise the image. Let $\mathbf{I}^c \in \mathbb{R}^{H \times W \times 3}.$0 denote the target image and $\mathbf{I}^c \in \mathbb{R}^{H \times W \times 3}.$1 the guided noisy image at diffusion step $\mathbf{I}^c \in \mathbb{R}^{H \times W \times 3}.$2. The reverse step estimates

$\mathbf{I}^c \in \mathbb{R}^{H \times W \times 3}.$3

The standard DDPM-style estimate of the clean image is

$\mathbf{I}^c \in \mathbb{R}^{H \times W \times 3}.$4

where

$\mathbf{I}^c \in \mathbb{R}^{H \times W \times 3}.$5

Prior work in diffusion-driven adaptation (DDA) uses only a predefined low-pass constraint:

$\mathbf{I}^c \in \mathbb{R}^{H \times W \times 3}.$6

FOCUS extends this by using both learned priors:

$\mathbf{I}^c \in \mathbb{R}^{H \times W \times 3}.$7

The conditioning signal therefore enforces agreement in both coarse semantic structure and fine boundary or detail structure during reverse denoising (Tjio et al., 20 Aug 2025). This is the operational role of Y-FPN within FOCUS: it does not itself perform the full denoising; rather, it parameterizes the frequency priors that guide denoising.

The distinction from fixed-prior approaches is conceptually important. In DDA, the prior is predefined and low-pass only. In FOCUS, the priors are learned from the corrupted input and include both low- and high-frequency terms. This suggests that Y-FPN is the mechanism by which FOCUS adapts the denoising objective to the spatially varying semantic demands of dense prediction.

5. Training regime and FrequencyMix

Y-FPN is trained separately before test time using clean images and a synthetic corruption pipeline called FrequencyMix. The training objective is

$\mathbf{I}^c \in \mathbb{R}^{H \times W \times 3}.$8

with the frequency reconstruction term

$\mathbf{I}^c \in \mathbb{R}^{H \times W \times 3}.$9

Here, $K \times K$0 is the Fourier transform, and the paper fixes

$K \times K$1

This loss trains Y-FPN to produce restorations that are close in pixel space, structurally similar, and spectrally aligned in frequency space (Tjio et al., 20 Aug 2025).

FrequencyMix is motivated by the need for Y-FPN to generalize across many corruption types with different spectral characteristics. For a single-channel image $K \times K$2, the Fourier decomposition is written as

$K \times K$3

with amplitude and phase

$K \times K$4

$K \times K$5

The method defines four perturbation types: $K \times K$6, $K \times K$7, $K \times K$8, and $K \times K$9. The amplitude spectrum is perturbed as

$K=3$0

with $K=3$1.

The perturbed amplitude is recombined with the original phase via inverse FFT to obtain an augmented image. FrequencyMix then randomly combines multiple frequency-corrupted versions of the same clean image: it samples mixing weights $K=3$2, creates several augmentation compositions, and forms a composite image

$K=3$3

where

$K=3$4

The algorithm explicitly samples combinations such as

$K=3$5

and randomly selects among them. The stated purpose is to expose Y-FPN to low-frequency corruption, high-frequency corruption, random spectral peaks, and uniform noise, thereby improving robustness across the broad corruption suite used at test time.

6. Empirical interpretation, forgetting mitigation, and computational profile

The empirical picture presented for Y-FPN is not that the module alone solves dense prediction under corruption, but that it becomes effective when used as a conditioning mechanism inside diffusion. The paper reports that Y-FPN only is not enough, that unconditioned diffusion can even hurt performance, and that conditioned diffusion with Y-FPN priors improves results substantially (Tjio et al., 20 Aug 2025). This is a useful corrective to a common simplification: the contribution is not merely “frequency prediction,” but frequency prediction coupled to reverse-process guidance.

Ablations further refine the role of the two branches. Low-frequency prior alone improves both segmentation and depth. Adding high-frequency prior further helps segmentation. For depth, high-frequency prior can slightly reduce performance in some settings, but for corruptions that destroy high-frequency content, combining both priors helps. This suggests that low-frequency preservation is the main driver, while high-frequency conditioning is task- and corruption-dependent, especially where boundaries and detail structure dominate prediction quality.

The paper also reports that using all FrequencyMix perturbations performs best, better than any single augmentation type, although AugHigh often helps most among single variants. Hyperparameter findings indicate that the best frequency loss weight $K=3$6 is around 0.1–0.2, the best diffusion timesteps are around 15–25, and stronger conditioning scale weights generally help.

With respect to continual adaptation, FOCUS is described as robust because it is not dependent on continual finetuning of the task model. It can also derive pseudo labels from FOCUS-denoised images for additional supervision, and these pseudo labels improve TENT, CoTTA, and especially DUSA, reducing the episodic-vs-continual gap associated with catastrophic forgetting. The paper states that FOCUS mitigates catastrophic forgetting in methods such as TENT and DUSA, and that even under limited, intermittent supervision with pseudo labels derived from the FOCUS denoised images, it mitigates catastrophic forgetting for recent model adaptation methods.

The computational tradeoff is explicit. Y-FPN has 27.5M parameters and is trained once offline. At test time, FOCUS uses a pretrained ImageNet diffusion model without finetuning it. Inference remains computationally expensive because diffusion denoising is slow, but the method is reported as lighter than DUSA and avoids the larger cost of finetuning both task and diffusion models. For FOCUS+DUSA, the paper notes reduced memory versus DUSA alone because the diffusion model itself is not finetuned in the combined setup.

Within the broader design of FOCUS, Y-FPN therefore occupies a specific and technically delimited role: it is the learned module that predicts spatially adaptive low- and high-frequency kernels from corrupted images; these kernels define pixel-wise frequency priors; and those priors condition reverse diffusion so that denoising preserves the semantic content required for dense prediction across corruption shifts (Tjio et al., 20 Aug 2025).