Towards Photorealistic and Efficient Bokeh Rendering via Diffusion Framework

Abstract: Existing mobile devices are constrained by compact optical designs, such as small apertures, which make it difficult to produce natural, optically realistic bokeh effects. Although recent learning-based methods have shown promising results, they still struggle with photos captured under high digital zoom levels, which often suffer from reduced resolution and loss of fine details. A naive solution is to enhance image quality before applying bokeh rendering, yet this two-stage pipeline reduces efficiency and introduces unnecessary error accumulation. To overcome these limitations, we propose MagicBokeh, a unified diffusion-based framework designed for high-quality and efficient bokeh rendering. Through an alternative training strategy and a focus-aware masked attention mechanism, our method jointly optimizes bokeh rendering and super-resolution, substantially improving both controllability and visual fidelity. Furthermore, we introduce degradation-aware depth module to enable more accurate depth estimation from low-quality inputs. Experimental results demonstrate that MagicBokeh efficiently produces photorealistic bokeh effects, particularly on real-world low-resolution images, paving the way for future advancements in bokeh rendering. Our code and models are available at https://github.com/vivoCameraResearch/MagicBokeh.

• The paper introduces MagicBokeh, a unified diffusion-based framework that simultaneously performs Real-ISR and bokeh rendering to enhance efficiency and visual fidelity.
• It leverages a high-quality feature extraction backbone, focus-aware mask attention, and degradation-aware depth estimation to address noise and artifact challenges in high-zoom imaging.
• Quantitative and qualitative results, including improved PSNR, SSIM, and user preference scores, validate its superior photorealistic output over traditional two-stage methods.

Photorealistic and Efficient Bokeh Rendering with MagicBokeh

Motivation and Problem Statement

Conventional mobile photography, due to compact optical hardware and small apertures, is fundamentally limited in producing optically realistic bokeh effects. While contemporary learned bokeh rendering methods have progressed, they remain inadequate for images captured under high digital zoom, which typically exhibit pronounced noise, artifact amplification, and a loss of fine structure. The prevailing two-stage pipeline—first, image restoration via super-resolution (Real-ISR), followed by bokeh rendering—suffers from error propagation, suboptimal visual quality, and inefficiency owing to multiple inference passes. This work introduces MagicBokeh, a unified diffusion-based framework designed to address these inherent limitations by performing Real-ISR and bokeh rendering simultaneously with enhanced controllability and computational efficiency.

Unified Diffusion-Based Framework

MagicBokeh leverages the representational power of diffusion models—specifically, architectures inspired by latent diffusion (à la Stable Diffusion)—to extract high-quality features from low-quality (LQ) inputs and render controllable bokeh effects in a unified forward pass.

The system consists of two principal modules: (1) a high-quality (HQ) feature extraction backbone that directly accepts LQ photos as input, dispensing with noise injection typical in generative modeling, and (2) a controllable bokeh rendering head, realized via a conditional ControlNet and further refined by a focus-aware masked attention mechanism. LoRA modules are injected for parameter-efficient finetuning during task-aligned training schedules.

Figure 1: The MagicBokeh framework alternates trainable submodules for Real-ISR and bokeh rendering. During inference, the DA depth model estimates disparity to guide bokeh synthesis from high-zoom LQ photos.

This design enables MagicBokeh to outperform cascaded pipelines in both speed and image fidelity, producing artifact-free, photorealistic bokeh under extreme conditions.

Alternative Training Strategy and Focus-aware Mask Attention

A central challenge of unifying Real-ISR and bokeh rendering objectives within a single backbone is their fundamentally conflicting gradients: Real-ISR seeks sharp subject reconstruction, while bokeh synthesis introduces out-of-focus blur selectively. End-to-end joint optimization degrades subject fidelity and overall rendering quality.

MagicBokeh resolves this via an alternative (cyclical) training scheme: it alternates between two fine-tuning phases, explicitly optimizing (a) the Real-ISR capacity (super-resolution of all-in-focus LQ–HQ pairs, with only dedicated LoRA layers updated), and (b) the bokeh rendering capacity (using ControlNet-conditional signals and updating only bokeh-rendering modules). This separation prevents task interference, leading to increased in-focus reconstruction accuracy and more consistent DoF effects.

Critical for region-selective control is the focus-aware mask attention (FAMA) module. This mechanism utilizes defocus maps derived from predicted disparity to impose binary spatial masks on attention activations, ensuring only background partitions are diffused with blur, while foregrounds remain sharp. This explicit spatial conditioning via modulated self-attention maintains semantic and perceptual separation between regions.

Figure 2: Compared with direct LR bokeh rendering and naive two-stage pipelines, MagicBokeh’s unified architecture synthesizes visually superior bokeh with a single pass and at lower computational cost.

Degradation-Aware Depth Estimation

Performance of depth priors degrades significantly in the presence of LQ, noisy, or compressed image data, which is typical in high digital zoom scenarios. MagicBokeh augments the baseline Depth Anything v2 model with a degradation-aware depth (DA depth) module trained via teacher–student self-feature distillation. The student is trained to maintain feature-level and output-level consistency with the teacher on HQ data, even when presented with synthetic LQ samples (simulated by Real-ESRGAN degradation).

The resulting DA depth module yields robust disparity maps that improve bokeh rendering accuracy for real, noisy mobile inputs. Quantitative evaluation confirms lower AbsRel error and higher $\delta_1$ on standard datasets—especially under severe degradation—compared to vanilla Depth Anything v2.

Figure 3: Training pipeline for the DA depth estimation module via feature-level distillation, improving robustness to severe input degradation.

Quantitative and Qualitative Results

MagicBokeh achieves state-of-the-art performance across both full-reference and no-reference evaluation metrics, as quantified on the EBB400-LQ synthetic benchmark and validated with human preference studies on real-world datasets. Metrics such as PSNR, SSIM, LPIPS, MUSIQ, FID, and user studies all decisively favor MagicBokeh over two-stage and prior SOTA single-stage approaches.

On EBB400-LQ, MagicBokeh attains 24.23 PSNR, 0.8623 SSIM, 0.2786 LPIPS, and the lowest FID (72.43) while being computationally most efficient—0.1062s per 512x512 inference, well ahead of all baselines. Notably, its qualitative outputs reveal improved edge transitions, more plausible blur extents, and enhanced realism particularly in challenging, noisy, or highly zoomed visual scenarios.

Figure 4: Qualitative comparison on EBB400-LQ, with MagicBokeh demonstrating marked improvements in edge fidelity and bokeh coherence relative to multi-stage competitors.

Furthermore, a 50-participant user study on real mobile images (iPhone 13 pro, 5–15× zoom, 4032×3024px) shows a pronounced preference for MagicBokeh outputs, attributing superior realism, natural DoF transitions, and subject-background separation.

Figure 5: User preference distribution confirms MagicBokeh’s perceived superiority over baseline methods in real-world, high-zoom bokeh rendering.

Ablation Analysis

Ablation experiments validate the necessity and synergy of the design components: the alternative training strategy, focus-aware mask attention, and the DA depth estimation module each positively impact both quantitative metrics and perceptual quality. Their removal leads to degradations such as increased artifacting (edge halos, unnatural background blur), subject blur, or faulty depth conditioning—supporting the architectural choices.

Figure 6: Visual ablation demonstrating the relative contributions of alternate training, FAMA, and DA depth; only the full model achieves artifact-free, realistic output across all regions.

Controllable DoF Effects and Refocusing

MagicBokeh supports precise control over both focal distance and aperture (blur intensity) post-inference by manipulating the input disparity/defocus map and associated blur parameters. This enables applications such as refocusing, progressivity in blur, and varied DoF simulation on a single LQ input.

Figure 7: Example showing dynamic adjustment of focus distance via defocus maps applied to the same LQ photo.

Figure 8: Example demonstrating progressive aperture scaling (1x to 3x blur) with consistent subject preservation.

Practical and Theoretical Implications

Practically, MagicBokeh eliminates the inefficiency, error accumulation, and fidelity loss endemic to two-stage image enhancement and bokeh pipelines, directly enabling real-time, high-fidelity bokeh rendering on degraded, high-zoom phone imagery in a single inference with explicit control. Theoretically, the success of region-conditioned attention and alternative training for conflicting objectives is likely extensible to other multi-task generative vision settings. The degradation-aware distillation for depth robustness presents a scalable paradigm for handling real-world noise and hardware artifacts.

Future Directions

Potential avenues include further integration with video bokeh and refocusing pipelines, domain-specific augmentation for cinematographic applications, and adaptation to other generative/ill-posed image transformation domains where high-level and low-level cues conflict.

Conclusion

MagicBokeh substantiates the utility of unified, diffusion-based architectures—with modular, region-aware conditioning and robust auxiliary estimators—for the task of photorealistic and efficient bokeh synthesis under realistic mobile imaging constraints. It achieves both practical deployment readiness and advances understanding of multi-task optimization and controllability in generative vision models.