LightsOut: Diffusion-based Outpainting for Enhanced Lens Flare Removal

Abstract: Lens flare significantly degrades image quality, impacting critical computer vision tasks like object detection and autonomous driving. Recent Single Image Flare Removal (SIFR) methods perform poorly when off-frame light sources are incomplete or absent. We propose LightsOut, a diffusion-based outpainting framework tailored to enhance SIFR by reconstructing off-frame light sources. Our method leverages a multitask regression module and LoRA fine-tuned diffusion model to ensure realistic and physically consistent outpainting results. Comprehensive experiments demonstrate LightsOut consistently boosts the performance of existing SIFR methods across challenging scenarios without additional retraining, serving as a universally applicable plug-and-play preprocessing solution. Project page: https://ray-1026.github.io/lightsout/

• The paper introduces a three-stage pipeline combining light source prediction, diffusion outpainting, and SIFR boosting that reconstructs missing illumination context to enhance flare removal.
• The methodology leverages LoRA-fine-tuned diffusion on Stable Diffusion v2 with explicit spatial conditioning and noise reinjection to ensure smooth and realistic outpainted regions.
• Improved metrics, such as increased PSNR and decreased LPIPS, validate the framework’s effectiveness over traditional SIFR models, setting new standards for lens flare removal.

Diffusion-based Outpainting for Enhanced Lens Flare Removal: An Analysis of "LightsOut"

Introduction

The "LightsOut" framework addresses a critical limitation in single-image flare removal (SIFR): the inability of existing models to effectively remove lens flare artifacts when off-frame light sources are incomplete or missing. The proposed solution leverages a diffusion-based outpainting pipeline, integrating a multitask regression module for light source parameter prediction and a LoRA-fine-tuned diffusion inpainting model, to reconstruct missing illumination context. This enables SIFR models to operate with enhanced context, significantly improving flare removal performance in challenging scenarios.

Figure 1: Motivation for outpainting incomplete off-frame light sources. SIFR methods degrade when off-frame sources are missing, motivating the need for outpainting.

Methodology

Three-Stage Pipeline

The LightsOut pipeline consists of three stages:

1. Light Source Prediction and Conditioning: A multitask regression module predicts the parameters (positions, radii, confidences) of off-frame or incomplete light sources. These parameters are used to render a light source mask, which serves as a conditioning input for subsequent stages.
2. Light Source Outpainting: A LoRA-fine-tuned diffusion-based inpainting model, conditioned on the predicted light source mask, generates outpainted regions that reconstruct missing light sources and associated flare artifacts. The model is trained on masked images and leverages both noise reinjection and RGB-space blending to ensure seamless and physically consistent results.
3. SIFR Boosting: The outpainted images are fed into existing SIFR models, which now benefit from complete illumination context, resulting in improved flare removal. The final output is obtained by extracting the original region from the SIFR-processed image.

   Figure 2: Overview of the three-stage pipeline: (a) light source prediction and conditioning, (b) light source outpainting, (c) SIFR boosting.

Multitask Regression for Light Source Prediction

The multitask regression module predicts $N$ sets of $(x, y, r)$ parameters and associated confidences for potential light sources. The architecture comprises a CNN feature extractor and two MLP heads for parameter and confidence estimation. The loss function combines a smooth L1 loss for parameter regression and a binary cross-entropy loss for confidence, with uncertainty-aware weighting to balance the two objectives. Bipartite matching ensures permutation invariance in the assignment of predicted and ground-truth sources.

Figure 3: The multitask regression module predicts physical parameters and confidences for light sources, enabling accurate mask generation.

Diffusion-based Outpainting with LoRA Fine-tuning

The outpainting model is based on Stable Diffusion v2, fine-tuned with LoRA for efficient adaptation. The model is conditioned on the predicted light source mask and a BLIP-2-derived text prompt. During inference, noise reinjection is employed to mitigate error accumulation and improve distributional alignment. RGB-space blending is used to avoid artifacts introduced by latent-space blending, ensuring smooth transitions at the boundaries of generated regions.

Conditioning Module

A dedicated conditioning module enforces the spatial placement of light sources in the generated content, using an L2 loss between the generated and predicted light source masks. This explicit conditioning is critical for physically plausible outpainting, especially in scenes with multiple or ambiguous light sources.

Experimental Results

Quantitative Evaluation

LightsOut demonstrates consistent improvements over both SIFR and diffusion-based outpainting baselines. For example, with Flare7K++ on real images without light sources, PSNR increases from 26.29 dB (direct input) to 28.41 dB (LightsOut), and LPIPS decreases from 0.0442 to 0.0397. The multitask regression module achieves higher mIoU scores (0.6310 real, 0.6619 synthetic) compared to a UNet baseline.

Qualitative Evaluation

Qualitative results show that SIFR models alone leave significant residual artifacts when off-frame sources are missing. Integrating LightsOut as a preprocessing step yields outputs that are visually closer to ground truth, with more realistic illumination and fewer artifacts.

Figure 4: Qualitative comparison of lens flare removal. LightsOut preprocessing enables SIFR models to produce results closer to ground truth in challenging scenarios.

Figure 5: Qualitative comparison of outpainting. LightsOut generates more realistic and physically consistent outpainted regions than standard diffusion-based methods.

Ablation Studies

Ablation experiments confirm the necessity of each component:

• Noise Reinjection: Improves visual coherence and reduces LPIPS.
• RGB vs. Latent Space Blending: RGB blending yields smoother transitions and better quantitative metrics.
• Conditioning Module: Essential for accurate localization and realism of outpainted light sources.
• Multitask Regression: Outperforms differentiable rendering and standard regression approaches for light source prediction.

  Figure 6: Ablation studies. (a) Noise reinjection enhances realism; (b) RGB blending outperforms latent blending; (c) Conditioning module improves accuracy and realism of generated light sources.

Implementation Considerations

• Computational Overhead: The outpainting stage introduces additional inference time, primarily due to the diffusion process. LoRA fine-tuning mitigates training cost but does not reduce inference latency.
• Plug-and-Play Integration: The pipeline is model-agnostic and can be inserted before any SIFR model without retraining the downstream model.
• Scalability: The approach is compatible with high-resolution images, subject to the memory and compute constraints of the underlying diffusion model.
• Limitations: The method does not address end-to-end optimization with SIFR models, and the multitask regression module assumes a fixed maximum number of light sources.

Implications and Future Directions

The LightsOut framework demonstrates that explicit reconstruction of missing illumination context is critical for robust flare removal. The integration of physically motivated conditioning into diffusion-based outpainting sets a precedent for similar approaches in other image restoration tasks where global context is essential. Future work may focus on reducing computational overhead via more efficient diffusion sampling or exploring joint end-to-end training with SIFR models. Additionally, extending the conditioning mechanism to handle more complex light source geometries or dynamic scenes could further enhance applicability.

Conclusion

LightsOut provides a principled and effective solution to the challenge of lens flare removal in images with incomplete or missing off-frame light sources. By combining multitask regression-based light source prediction, LoRA-fine-tuned diffusion outpainting, and explicit conditioning, the framework significantly enhances the performance of existing SIFR models. The approach is modular, generalizable, and demonstrates strong empirical gains, establishing a new standard for preprocessing in flare removal pipelines.