BokehDiff: Diffusion-Based Bokeh Control

• BokehDiff is a family of diffusion-based methods designed for realistic bokeh synthesis, defocus manipulation, and lens-blur rendering without relying on extra depth or segmentation cues.
• It employs stage-conditioned latent diffusion and explicit physical parameters to control focus, mitigate artifacts, and enhance semantic integrity in both text-to-image and image-to-image tasks.
• These approaches enable applications in real-image editing and video refocusing, offering physics-guided lens blur simulation and consistent defocus control across varied photography scenarios.

Searching arXiv for papers on BokehDiff and diffusion-based bokeh rendering. BokehDiff denotes a family of diffusion-based approaches to bokeh synthesis, defocus manipulation, and lens-blur rendering. The term has been used in multiple, partly overlapping senses: as a stage-conditioned latent-diffusion pipeline for semantic bokeh generation in text-to-image and image-to-image synthesis; as a scene-consistent defocus-control framework that explicitly conditions a diffusion model on a physical blur parameter; and as a one-step neural lens-blur renderer that injects optics-inspired constraints into self-attention. Across these usages, the common objective is to control depth-of-field or bokeh while avoiding the limitations of prompt-only approximations, depth-discontinuity artifacts, or per-prompt optimization (Deng et al., 2023, Fortes et al., 11 Mar 2025, Zhu et al., 24 Jul 2025).

1. Terminological scope and research lineage

In the recent literature, “BokehDiff” does not denote a single canonical architecture. In the GBSD work, it denotes a diffusion-based pipeline for semantic bokeh generation via stage-conditioned latent diffusion, with the stated objective of being the first generative text-to-image approach for photorealistic bokeh style (Deng et al., 2023). In later work, “Bokeh Diffusion” refers to scene-consistent defocus blur control in text-to-image diffusion models through an explicit physical defocus blur parameter, a hybrid training pipeline, and grounded self-attention (Fortes et al., 11 Mar 2025). A further usage appears in “BokehDiff: Neural Lens Blur with One-Step Diffusion,” where the term identifies a single-image neural lens-blur renderer built around physics-inspired self-attention and one-step inference (Zhu et al., 24 Jul 2025).

These usages emerged against a shared background. Classical post-process bokeh rendering typically requires depth maps, segmentation masks, or 3D priors, and the cited failure modes include depth discontinuity artifacts, color bleeding, edge halos, and restrictions to the styles present in training data. Prompt-based diffusion alternatives can synthesize artistic styles, but the literature repeatedly identifies global scene drift, coarse approximations of lens blur, and weak control over focus placement as persistent problems (Deng et al., 2023, Fortes et al., 11 Mar 2025). This suggests that “BokehDiff” now functions both as a proper method name and as a broader label for diffusion-based bokeh control.

2. Stage-conditioned latent diffusion and semantic bokeh

GBSD formulates BokehDiff as a two-stage conditioning regime on top of a latent diffusion model, specifically Stable Diffusion v1-4. The architecture comprises a text encoder $\tau_\theta$, a VAE encoder $\epsilon(x)$ and decoder $D(z)$, and a latent-space UNet denoiser $\epsilon_\theta(z_t, t, \tau_\theta(y))$. The latent diffusion training objective follows the standard LDM “simple” loss,

$$L_{LDM} = E_{z, y, \epsilon\sim N(0,1), t} [\|\epsilon - \epsilon_\theta(z_t, t, \tau_\theta(y))\|^2_2].$$

The distinctive mechanism is the split of denoising into a global-layout phase and a focus-and-bokeh phase (Deng et al., 2023).

In stage 1, a global prompt $y_{\text{global}}$ defines scene semantics, object placement, shape, and color. The model allocates a fraction $\sigma \in (0,1)$ of the total timesteps $T$ to this phase and produces an intermediate latent

$$z_{\sigma\cdot T} = g(\tau_\theta(y_{\text{global}}), z_T, \sigma).$$

In stage 2, a local prompt $y_{\text{local}}$ specifies the object to remain in focus. GBSD linearly interpolates the prompt embeddings as

$\epsilon(x)$0

then runs the remaining denoising steps under $\epsilon(x)$1. The paper states that $\epsilon(x)$2–$\epsilon(x)$3 retains object identity, whereas too small a $\epsilon(x)$4 leads to semantic mixing; at $\epsilon(x)$5, carrots–rabbit feature mixing is reported (Deng et al., 2023).

The semantic effect follows directly from the stage schedule. The object emphasized by $\epsilon(x)$6 receives refined detail in stage 2, while non-mentioned objects drift toward a bokeh-like blur because high-frequency details are not reinforced. The method requires no retraining, no fine-tuning per prompt, no masks, and no cross-attention edits, and it operates in both text-to-image and image-to-image modes. The reported experimental setup uses NVIDIA V100 GPUs, output resolution $\epsilon(x)$7, batch size 4, seed 42, DDIM sampling steps 50, total timesteps $\epsilon(x)$8, and guidance scale 15 (Deng et al., 2023).

Quantitatively, GBSD reports that in a text-to-image “carrots focus” case it increased in-focus sharpness on carrot segments with $\epsilon(x)$9 and $D(z)$0, corresponding to $D(z)$1 and $D(z)$2 improvements over the baseline, while reducing sharpness on the bunny. In an image-to-image focus-shift example with global prompt “A cute rabbit stands with carrots with green leaf” and local prompt “carrots with green leaf,” it sharpened the carrots by 73% in VoL and 93% in Brenner relative to the input image, reported also as $D(z)$3 and $D(z)$4 increases, while the rabbit exhibited lower VoL and Brenner values, consistent with bokeh (Deng et al., 2023).

3. Explicit physical conditioning and scene-consistent defocus control

“Bokeh Diffusion: Defocus Blur Control in Text-to-Image Diffusion Models” reframes BokehDiff around explicit physical conditioning. Rather than relying on prompt phrases such as aperture descriptors, it introduces a scalar defocus parameter aligned to a blur gain $D(z)$5, and it seeks scene-consistent bokeh: blur strength should change while scene composition and semantic content remain intact (Fortes et al., 11 Mar 2025). The optical parameterization is grounded in thin-lens relations, including the CoC diameter

$D(z)$6

and the disparity-based defocus radius

$D(z)$7

The model is trained with a hybrid pipeline that alternates between real in-the-wild images and synthetic focus–defocus pairs. The base corpus is a curated subset of Megalith-10M; for each selected photograph the pipeline estimates metric depth with Depth Pro, foreground masks with RMBG-2.0, extracts EXIF metadata, and generates captions. Nearly all-in-focus images, stated as about 10% of the 15K-image corpus, are synthetically augmented with BokehMe to sample multiple blur intensities on the same scene. Real and synthetic batches are sampled with equal probability, and $D(z)$8 is aligned across the two sources through a supplementary mapping (Fortes et al., 11 Mar 2025).

Architecturally, the model adds a lightweight bokeh cross-attention adapter in deeper decoder cross-attention layers of a latent UNet and preserves the original text-based keys and values in frozen form. A separate grounded self-attention mechanism is activated during contrastive paired training. Its stated role is to use a pivot image to anchor scene structure while allowing bidirectional blur adjustments. The paper also introduces localized cross-attention regularization toward background regions via downsampled foreground masks (Fortes et al., 11 Mar 2025).

This formulation extends beyond generation to grounded editing and real-image inversion. At inference, a single scalar defocus parameter governs bokeh strength; larger values yield stronger blur. The method supports unbound generation, pivot-guided grounded generation, and real-image editing via inversion. On a benchmark of 20 prompts and 5 bokeh levels, it reports $D(z)$9, $\epsilon_\theta(z_t, t, \tau_\theta(y))$0, and a GPT-4o scene-consistency rating of 94.75, whereas prompt-based SD1.5, SD3.5, and FLUX baselines exhibit negative or weak blur-trend correlation and materially worse consistency (Fortes et al., 11 Mar 2025).

4. Physics-inspired attention and one-step neural lens blur

“BokehDiff: Neural Lens Blur with One-Step Diffusion” moves the problem from text-conditioned generation toward single-image lens-blur rendering. The method starts from the observation that depth-guided post-process blur remains bounded by depth errors, especially at fine structures and depth discontinuities, while iterative diffusion editing introduces stochasticity and structural drift. Its answer is a one-step adaptation of a pretrained SDXL latent diffusion model combined with a physics-inspired self-attention module, abbreviated PISA (Zhu et al., 24 Jul 2025).

The optical prior is encoded through a depth-dependent circle of confusion. The paper derives

$\epsilon_\theta(z_t, t, \tau_\theta(y))$1

and in practice uses a proportional disparity-based approximation for attention masking,

$\epsilon_\theta(z_t, t, \tau_\theta(y))$2

Attention is then modified in three ways. First, normalization is performed across queries so that each source pixel contributes a fixed total energy:

$\epsilon_\theta(z_t, t, \tau_\theta(y))$3

Second, a soft CoC-constrained support mask

$\epsilon_\theta(z_t, t, \tau_\theta(y))$4

restricts interactions to physically plausible neighborhoods. Third, a visibility term $\epsilon_\theta(z_t, t, \tau_\theta(y))$5 gates attention so that background light cannot bleed through nearer foreground structures (Zhu et al., 24 Jul 2025).

Inference is deliberately non-iterative. The all-in-focus image is encoded into a latent $\epsilon_\theta(z_t, t, \tau_\theta(y))$6 with fixed timestep $\epsilon_\theta(z_t, t, \tau_\theta(y))$7 and no added Gaussian noise, after which a single denoising step produces $\epsilon_\theta(z_t, t, \tau_\theta(y))$8 and the final image $\epsilon_\theta(z_t, t, \tau_\theta(y))$9. Only low-rank adapters are trained: LoRA of rank 8 in the UNet downsampling blocks and in the middle/output of the encoder. The loss combines pixel-space MSE, VGG perceptual loss, a multi-scale edge loss, and an adversarial term, with $$L_{LDM} = E_{z, y, \epsilon\sim N(0,1), t} [\|\epsilon - \epsilon_\theta(z_t, t, \tau_\theta(y))\|^2_2].$$0, $$L_{LDM} = E_{z, y, \epsilon\sim N(0,1), t} [\|\epsilon - \epsilon_\theta(z_t, t, \tau_\theta(y))\|^2_2].$$1, $$L_{LDM} = E_{z, y, \epsilon\sim N(0,1), t} [\|\epsilon - \epsilon_\theta(z_t, t, \tau_\theta(y))\|^2_2].$$2, and $$L_{LDM} = E_{z, y, \epsilon\sim N(0,1), t} [\|\epsilon - \epsilon_\theta(z_t, t, \tau_\theta(y))\|^2_2].$$3 (Zhu et al., 24 Jul 2025).

To compensate for the lack of scalable paired data, the paper synthesizes paired all-in-focus and bokeh examples by generating photorealistic transparent foregrounds with diffusion models, compositing them with real small-aperture backgrounds, and ray tracing layered scenes. The reported benchmarks include EBB Val294, BLB Level 5, and SynBokeh300. On EBB Val294, exposure-aligned, BokehDiff reports PSNR 24.652, SSIM 0.8357, DISTS 0.1155, and LPIPS 0.3737; on BLB Level 5 it reports PSNR 36.798, SSIM 0.9814, and LPIPS 0.0888; on SynBokeh300 it reports PSNR 34.165, SSIM 0.9784, DISTS 0.0433, and LPIPS 0.1119. Average runtime is stated as 3.974 s per image on RTX A6000, and user studies report top scores in accuracy, authenticity, and preference (Zhu et al., 24 Jul 2025).

5. Control variables, domains, and later extensions

A central distinction inside the BokehDiff literature is the control variable. Early semantic approaches manipulate bokeh through prompt staging; later systems expose camera-like quantities such as a physical blur scalar, a focus point, a focal-plane disparity, or a blur-strength parameter. The progression is from semantic emphasis toward explicit refocusing and temporally coherent depth-aware rendering (Deng et al., 2023, Fortes et al., 11 Mar 2025, Wang et al., 30 Sep 2025, Yang et al., 27 May 2025).

Method	Control variables	Domain
GBSD	$$L_{LDM} = E_{z, y, \epsilon\sim N(0,1), t} [\|\epsilon - \epsilon_\theta(z_t, t, \tau_\theta(y))\|^2_2].$$4, $$L_{LDM} = E_{z, y, \epsilon\sim N(0,1), t} [\|\epsilon - \epsilon_\theta(z_t, t, \tau_\theta(y))\|^2_2].$$5, $$L_{LDM} = E_{z, y, \epsilon\sim N(0,1), t} [\|\epsilon - \epsilon_\theta(z_t, t, \tau_\theta(y))\|^2_2].$$6, $$L_{LDM} = E_{z, y, \epsilon\sim N(0,1), t} [\|\epsilon - \epsilon_\theta(z_t, t, \tau_\theta(y))\|^2_2].$$7	Text-to-image and image-to-image
Bokeh Diffusion	Scalar defocus parameter aligned to $$L_{LDM} = E_{z, y, \epsilon\sim N(0,1), t} [\|\epsilon - \epsilon_\theta(z_t, t, \tau_\theta(y))\|^2_2].$$8; optional pivot image	Text-to-image and real-image editing via inversion
DiffCamera	Focus point coordinates $$L_{LDM} = E_{z, y, \epsilon\sim N(0,1), t} [\|\epsilon - \epsilon_\theta(z_t, t, \tau_\theta(y))\|^2_2].$$9 and bokeh level $y_{\text{global}}$0; optional depth $y_{\text{global}}$1	Arbitrary refocusing on images
Any-to-Bokeh	Focal plane disparity $y_{\text{global}}$2, blur strength $y_{\text{global}}$3, focal plane-adapted MPI mask $y_{\text{global}}$4	One-step video bokeh

DiffCamera is the clearest move toward arbitrary post-capture refocusing. It uses a diffusion transformer trained with rectified flow, conditions on a new focus point $y_{\text{global}}$5 and a blur level $y_{\text{global}}$6, and supplements the basic flow loss with a stacking constraint derived from focus stacking. The benchmark comprises 150 scenes and evaluates refocus, add-bokeh, and remove-bokeh tasks. Reported results include, for refocus versus GPT-4o editing, CLIP-I 0.954 versus 0.859, MAE 0.025 versus 0.138, and CLIP-IQA 0.834 versus 0.724; for add bokeh versus GPT-4o, CLIP-I 0.969 versus 0.792, MAE 0.022 versus 0.087, LVCorr 0.920, and CLIP-IQA 0.857 versus 0.567 (Wang et al., 30 Sep 2025).

Any-to-Bokeh extends the same design space to video. It builds a one-step video diffusion model on top of Stable Video Diffusion, conditions on a normalized disparity difference to the focal plane, a scalar blur strength $y_{\text{global}}$7, and a focal plane-adapted MPI mask, and introduces a progressively widening disparity-interval sampling function

$y_{\text{global}}$8

Its reported quantitative results on video benchmarks are FD 0.431, RM 0.007, VFID-I 1.479, FVD 9.005, SSIM 0.974, PSNR 38.899, and Time 0.363, with substantial improvements over image-based baselines extended to video (Yang et al., 27 May 2025). This suggests that later BokehDiff-related systems increasingly treat bokeh as a controllable camera effect rather than a purely stylistic prompt attribute.

6. Limitations, misconceptions, and open directions

A recurring misconception is that BokehDiff is the name of a single stable benchmark method. The literature itself is more heterogeneous. GBSD explicitly operationalizes BokehDiff as staged semantic conditioning, whereas Any-to-Bokeh states that the term “BokehDiff” itself is not used as the name of that work, and MagicBokeh presents BokehDiff only as a baseline in two-stage pipelines (Deng et al., 2023, Yang et al., 27 May 2025, Shi et al., 8 May 2026). The term therefore names a research direction as much as a specific model.

The limitations also differ by formulation. GBSD lacks explicit optics controls such as aperture shape, focal plane, or circle-of-confusion parameterization; success depends on a well-chosen local prompt, and small $y_{\text{global}}$9 induces semantic mixing (Deng et al., 2023). Bokeh Diffusion can still suffer from parameter estimation noise in focus distance, and very large blur changes may induce subtle shifts in edges or contours despite grounding (Fortes et al., 11 Mar 2025). The one-step neural lens-blur BokehDiff remains dependent on a reasonably accurate disparity map and assumes a thin-lens, paraxial regime; severe depth errors, extreme translucency, and reflective materials remain challenging (Zhu et al., 24 Jul 2025). DiffCamera is currently trained at $\sigma \in (0,1)$0 and $\sigma \in (0,1)$1, with other aspect ratios not supported, and Any-to-Bokeh still struggles when disparity maps are severely wrong (Wang et al., 30 Sep 2025, Yang et al., 27 May 2025).

Open directions follow directly from these limitations. GBSD proposes adding optics-aware priors, depth estimation for more precise per-pixel focus, or cross-attention masking while retaining its simplicity (Deng et al., 2023). DiffCamera’s stacking constraint suggests a route toward more physically grounded controllability in single-image refocusing (Wang et al., 30 Sep 2025). Any-to-Bokeh points toward video refocusing with temporal constraints and longer time-series memory (Yang et al., 27 May 2025). MagicBokeh broadens the scope further by jointly optimizing bokeh rendering and super-resolution with focus-aware masked attention and a degradation-aware depth module for low-quality, high-zoom mobile inputs (Shi et al., 8 May 2026). Taken together, these works position BokehDiff as a transition from semantic bokeh generation toward explicit, physically informed, and increasingly deployment-oriented control of defocus blur.