---
title: 'FBSDiff: Frequency-Guided I2I Translation'
url: https://www.emergentmind.com/topics/fbsdiff
type: topic
---

# FBSDiff: Frequency-Guided I2I Translation

FBSDiff is an algorithmic framework for text-driven image-to-image (I2I) translation based on plug-and-play frequency band substitution of latent diffusion features. It enables highly controllable manipulation of image synthesis using a frozen, pre-trained text-to-image (T2I) diffusion model by dynamically substituting only selected frequency bands from a reference image’s diffusion features into the generation trajectory. This approach allows operators to fine-tune the transfer of appearance, layout, and contour information, or interpolate control strength by modulating the bandwidth of the substituted frequencies. FBSDiff and its accelerated extension FBSDiff++ offer versatile, training-free pipelines for state-of-the-art text-conditioned I2I editing, supporting global or localized manipulation, arbitrary input resolutions, and both structure-holding and style-transferring operations [2408.00998][2601.19115].

## 1. Frequency-Band Decomposition and Motivation

In conventional T2I or I2I translation with diffusion models, semantic content, appearance, and structural detail are entangled in spatial-domain latent representations, limiting control over which attributes from a reference image are transferred. FBSDiff introduces a DCT-based frequency separation of latent features, enabling explicit control:

- **Low-frequency coefficients** (top-left of DCT spectrum): encode appearance, color distributions, and broad layout statistics.
- **Mid-frequency coefficients**: encode intermediate-scale layout or region arrangements.
- **High-frequency coefficients** (bottom-right): encode edge information, contours, and fine detail.

By selectively substituting only a desired frequency band at each denoising step, FBSDiff achieves plug-and-play, fine-grained control of guiding factors—appearance, layout, or contour—without modifying model weights or performing any additional training or tuning.

## 2. Mathematical and Algorithmic Formulation

Let $x$ denote the reference image with latent encoding $z_0=E(x)$. During generation, the core Frequency Band Substitution (FBS) layer operates as follows:

\[
\tilde z_t' = \mathrm{IDCT}\Bigl(\mathrm{DCT}(\hat z_t) \cdot \mathit{Mask}_* + \mathrm{DCT}(\tilde z_t) \cdot (1 - \mathit{Mask}_*)\Bigr)
\]
where $\hat z_t$ and $\tilde z_t$ are the reference and generation latent features at timestep $t$, DCT and IDCT are channelwise 2D or cascaded 1D discrete cosine/inverse cosine transforms, and $\mathit{Mask}_*$ denotes a binary mask selecting a frequency band by index sum ($x+y$ for 2D). Mask types include:
- Low-pass: $\mathit{Mask}_{lp}(x,y)=1$ if $x+y \leq th_{lp}$
- High-pass: $\mathit{Mask}_{hp}(x,y)=1$ if $x+y > th_{hp}$
- Mid-pass: $\mathit{Mask}_{mp}(x,y)=1$ if $th_{mp1} < x+y \leq th_{mp2}$

In FBSDiff++, static 2D-DCT is replaced with cascaded 1D-DCT along width and height, with percentile-based thresholds (e.g., $pt_{lp}$) ensuring the method operates on arbitrary image shapes and aspect ratios.

The I2I pipeline consists of:
1. **Inversion:** DDIM inversion of the reference image to noise
2. **Reconstruction:** (FBSDiff only) reconstruction to cache the feature trajectory $\{\hat z_t\}$
3. **Text-guided sampling:** generation from random noise under target text
4. **Dynamic FBS:** at each step, substitute the chosen frequency band after denoising during the “calibration” phase

FBSDiff++ caches the inversion trajectory and entirely eliminates explicit reconstruction, providing significant acceleration.

## 3. Plug-and-Play Integration and Control Mechanisms

FBSDiff requires only standard UNet denoiser invocations (no model updates). During the initial $\lambda T$ sampling steps (calibration phase), FBS is repeatedly applied; text-only denoising resumes thereafter. The *guiding factor*—appearance, layout, or contour—is chosen via mask type; *guiding intensity* is adjusted by varying the mask thresholds (frequency band widths), yielding continuous interpolation from weak to strong attribute transfer.

FBSDiff++ extends this paradigm with:
- **AdaFBS:** automatic percentile-based mask scaling for arbitrary $H\times W$
- **Localized control:** region masks allow spatially restricted editing
- **Style-specific content creation:** random spatial transformations (spatial transformation pool, or STP) “destroy” explicit structure in the reference prior to low-frequency substitution, enabling style transfer without layout copying

## 4. Empirical Performance and Comparative Evaluation

Extensive experiments on LAION-Mini and benchmark protocols demonstrate:

- **Qualitative versatility:** By switching among band types, FBSDiff/++ can produce appearance-matching, layout-constrained, or contour-preserving edits, surpassing contemporaneous approaches such as Null-text Inversion, Plug-and-Play, Pix2Pix-zero, InstructPix2Pix, and StyleDiff in visual fidelity and attribute disentanglement.
- **Diversity:** Multiple distinct outputs for the same reference/text are produced via stochastic resampling of initial noise, in contrast to inversion-locked methods.
- **Quantitative superiority:** FBSDiff/++ achieves top-4 ranks on Structure Sim (DINO), LPIPS, AdaIN style loss, CLIP-similarity, and aesthetic scores under both derivative and style-transfer tasks. FBSDiff++ reaches 0.965 Structure Sim and leads in several trade-off metrics.
- **Efficiency:** Architecture streamlining enables FBSDiff++ to achieve an 8.9$\times$ speedup over FBSDiff and 7–8$\times$ over other state-of-the-art methods, with total processing time of 9.6s (A100, SD v1.5, 50 steps).
- **User study validation:** Among 70 raters and 20 models, FBSDiff++ attained the highest “Excellent/Optimal” rating proportion.

| Method        | Inversion | Sampling | Total Time   |
|---------------|-----------|----------|-------------|
| PT-Inv        | 42 s      | 30 s     | 72 s        |
| StyleDiff     | 40 s      | 38 s     | 78 s        |
| FBSDiff       | 77 s      | 8 s      | 85 s        |
| FBSDiff++     | 3.5 s     | 6.1 s    | 9.6 s       |

## 5. Ablation, Limitations, and Future Research

Ablations indicate that:
- Replacing stepwise dynamic substitution with a one-shot substitution at $t=\lambda T$ results in severe artifacts.
- Full-spectrum substitution collapses text fidelity; selective (partial) band substitution is critical.
- STP is necessary for structure-decoupled style transfer.

Identified limitations include:
- Hyperparameter tuning (frequency thresholds, $\lambda$) is currently manual; automated controllers could improve usability.
- FBSDiff applies global frequency masks, lacking spatially variant control.
- In cases of strong semantic mismatch between reference and text, some blending artifacts can appear.
- The current formulation is channelwise and DCT-based; extensions to other frequency representations or pixel-space diffusion are open research directions.

## 6. Practical Considerations and Usage Guidelines

FBSDiff/++ is implemented in Python/PyTorch leveraging HuggingFace diffusers (Stable Diffusion v1.5) and scipy for DCT computations. Key parameters:
- Number of steps $T=50$, CFG scale $\omega=7.5$, calibration ratio $\lambda=0.5$
- Frequency thresholds or percentiles tuned for desired attribute transfer
- Region masks and STP available for local edits and style-only generation, respectively
- The method is resolution-agnostic in FBSDiff++, generalizing to arbitrary image sizes without retraining.

A typical pipeline involves encoding the source image, DDIM inversion, caching latent features, text-guided sampling with AdaFBS at each step, and decoding the output. Masking, style-only, or other functionalities are achieved by simple modular additions.

## 7. Significance and Impact

FBSDiff introduces a frequency-domain methodology for factor-controllable text-driven I2I by exploiting the spectral separation of appearance, layout, and contours in DCT space. Its training-free, model-agnostic, and highly efficient implementation establishes a new class of practical, tunable I2I pipelines for both research and application contexts, providing a new avenue for controllable generative image editing with strong empirical performance across tasks and benchmarks [2408.00998][2601.19115].

Source: https://www.emergentmind.com/topics/fbsdiff