Physically Guided Controllable Bokeh Generator

• The paper presents a physically guided framework that models optical blur using thin-lens equations and differentiable convolution to decouple scene content from defocus effects.
• It integrates an all-in-focus generator, monocular depth estimation, transformer-based focus prediction, and EXIF metadata conditioning to enable interactive bokeh control.
• Quantitative benchmarks, including 93.9% blur monotonicity and 92.3% content consistency, validate the system's superior photorealism over traditional data-driven methods.

A physically guided controllable bokeh generator is a computational framework designed to synthesize or edit images (or video) with user-controlled, optically accurate defocus blur ("bokeh"), closely modeling the actual image formation of real camera lenses. Such systems explicitly parameterize lens properties—aperture, focal length, and focus distance—and use differentiable, physics-motivated representations to reproduce shallow depth-of-field effects that are indistinguishable from true photographic bokeh. Unlike prompt-tuned diffusion models or empirically learned CNNs, physically guided generators disentangle scene content from lens blur, enabling faithful, interactive manipulation of defocus without altering semantic content.

1. Physical Optics and Thin-Lens Blur Modeling

Controllable bokeh synthesis is grounded in thin-lens optics, which dictates how lens parameters shape the spatial distribution of defocus blur (circle of confusion, CoC). The CoC diameter at pixel location $(i,j)$, for a scene point at depth $d_{i,j}$ when the camera is focused at $D$ with focal length $f$ and f-number (aperture) $N$, is modeled as: $$coc_{i,j} = \frac{|\,d_{i,j}-D\,|}{d_{i,j}} \frac{f^2}{N(f_s D - f)}$$ where $f_s$ is a learned depth-to-focus scaling parameter to account for scale mismatches between monocular depth prediction and metric distance. This formula is a direct consequence of the paraxial lens equation and encapsulates the dependency of blur strength on both lens settings and scene geometry (Shrivastava et al., 7 Oct 2025).

At each pixel, the CoC is used to construct a normalized, soft-edged disk kernel. The physically motivated convolution of the all-in-focus image with these spatially varying kernels yields the very shallow depth-of-field appearance characteristic of bokeh.

2. Architecture: Decoupling Scene, Depth, Focus, and Optics

A representative pipeline for physically guided controllable bokeh generation (e.g., Fine-grained Defocus Blur Control (Shrivastava et al., 7 Oct 2025)) consists of the following four stages:

1. All-in-Focus Generation: Leveraging a distilled Stable Diffusion XL (SDXL) model, a student generator $G_\theta$ outputs an all-in-focus RGB image conditioned on a text prompt and noise vector. Knowledge distillation with Distribution Matching Distillation (DMD) and GAN loss ensures realism and content faithfulness.
2. Monocular Depth Estimation: A frozen metric depth estimator (e.g., Metric3Dv2) predicts dense depth from the generated image, decoupling geometric inference from generative content synthesis.
3. Focus Distance Prediction via Transformer: A fine-tuned Visual Saliency Transformer (VST) computes a weighted-average focus distance $D$, fusing per-pixel saliency $s$ and depth $\mathbf{d}$, with additional scaling from a learned MLP head.
4. Differentiable Lens Blur Synthesis: Given per-pixel CoC, a spatially varying convolution blurs the image with a disk kernel. FFT-based implementation and smooth soft edges enable differentiability, allowing gradients to flow from the final output through the blur, focus, depth, and generative stages (Shrivastava et al., 7 Oct 2025).

This pipeline facilitates fully end-to-end training, supporting joint optimization and weak supervision on unpaired deep/shallow DoF datasets, and enables test-time control over any lens or scene parameter.

3. Explicit Camera Metadata Conditioning and User Interface

Integration of real camera metadata is pivotal for both veridical bokeh rendering and controllability. Key EXIF parameters (aperture $N$, focal length $f$) are:

• Encoded via sinusoidal positional embeddings, concatenated, and linearly projected to produce the "EXIF embedding."
• Used to modulate timestep embeddings during image generation and to directly parameterize blur kernel computations (Shrivastava et al., 7 Oct 2025).

During interactive inference, users specify the text prompt, EXIF-derived or manual aperture and focal length, and optionally override focus distance. Sliders for $N$ (f-stop) and $D$ (focus) directly map to blur strength and in-focus-plane location. This configuration enables real-time, intuitive bokeh tuning on synthetically generated scenes.

4. Differentiable Blur and Backpropagation for End-to-End Learning

Differentiable lens blur is essential for propagating photometric and semantic losses through all pipeline stages. The per-pixel blur kernel

$$h_{i,j}(u,v) = \frac{\exp\!\bigl(-\|[u,v]\|^2 / (2\sigma_{i,j}^2)\bigr)}{\sum_{u',v'} \exp\!\bigl(-\|[u',v']\|^2 / (2\sigma_{i,j}^2)\bigr)}$$

where $\sigma_{i,j} \propto r_{i,j} = coc_{i,j}/2$, is applied in a spatially varying fashion using FFT-based convolutions. This approach ensures that both in-focus and out-of-focus regions are synthesized with authentic spatial variation and that the system can be trained under weak supervision.

Backpropagation through this lens blur model allows the learning of not only blur parameters but also the generator, depth, and focus modules in tandem, resulting in sharper subject preservation and realistic edge transitions.

5. Loss Functions, Training Strategy, and Dataset Utilization

Training utilizes heterogeneous datasets:

• Deep DoF (all-in-focus) images supervise the generator to maintain sharp scene depiction.
• Shallow DoF (bokeh-rich) images supervise the entire pipeline to replicate authentic lens blur.

The total loss combines DMD2 for denoising, KL divergence, GAN-based realism for both "deep" and "shallow" paths, and a heavily weighted Huber loss for focus prediction. Typical weights are $\lambda_1=1,\;\lambda_2=1,\;\lambda_3=200$ (Shrivastava et al., 7 Oct 2025). No direct paired supervision is required between sharp and bokeh images per scene.

This weakly supervised, multi-branch loss structure exploits the physical invariance of optics and the robust priors of generative diffusion.

6. Quantitative Evaluation, Benchmarks, and Comparative Analysis

Comprehensive evaluation examines:

• Blur Monotonicity: Fraction of monotonic decreases in image energy as aperture is closed (desired: near 100%).
• Content Consistency: Semantic segmentation consistency across variable aperture (desired: near 100%).
• LPIPS: Perceptual similarity for bokeh transitions (lower is better).
• FID: Generative realism scores on both Deep and Shallow DoF (lower is better).

On 10K test scenes, the analyzed physically guided generator (with TAF lens) achieved:

• Blur Monotonicity 93.9%
• Content Consistency 92.3%
• LPIPS 0.0064
• FID$_\mathrm{Deep}$=13.24, FID$_\mathrm{Shallow}$=16.69

Full ablation and baseline comparisons demonstrate clearly superior control-fidelity and scene permanence compared to prompt-conditioning and engineered or plug-in blur modules (Shrivastava et al., 7 Oct 2025).

7. Implementation Details, Limitations, and Future Directions

Training is performed on 1.5M deep/shallow DoF scenes at 1024×1024 with 16 A100 GPUs over 2 days, using AdamW ($5 \times 10^{-7}$, batch size 1). The depth estimator and VST are frozen, except for the focus distance heads.

Limitations include:

• Dependence on monocular depth accuracy, with potential for unwanted background blur if estimator is biased.
• Present support for circular CoC only; arbitrary aperture shapes (hexagon, star, etc.) require replacing the disk kernel.
• Focus distance supervision is weak; direct RGB-D plus explicit focus sensors would further calibrate scale prediction.

Explicit future work includes extending to arbitrary PSFs, refining with higher-quality depth/focus ground truth, and supporting more complex, non-circular lens artifacts.

---

In summary, physically guided controllable bokeh generators fuse thin-lens physics, robust monocular depth prediction, transformer-based focus saliency, EXIF-conditioned diffusion, and end-to-end differentiable blur into a unified pipeline. This architecture affords interactive, photorealistic bokeh control—precise modulation of depth-of-field and focus—while rigidly preserving scene content, outperforming purely prompt-based or data-driven approximations in both fidelity and usability (Shrivastava et al., 7 Oct 2025).