Virtual Highlight Synthesis

• Virtual Highlight Synthesis is a method that generates synthetic specular highlights on RGB images using physically-based rendering with Blinn–Phong and Fresnel modulation.
• It leverages monocular geometry estimation and randomizes point light sources to create pseudo paired data without explicit ground-truth diffuse-specular pairs.
• This approach enables the training of highlight removal networks that achieve state-of-the-art SSIM and reduced luminance suppression in both natural and surgical imagery.

Virtual Highlight Synthesis refers to a physically-based, data-driven approach for generating realistic synthetic specular highlights on arbitrary RGB images. This methodology constructs pseudo paired data—comprising clean “diffuse” images, their synthetic specular augmentations, and corresponding highlight masks—to enable highlight removal networks to be trained without explicit diffuse-specular ground-truth pairs. The technique plays a critical supervisory role in state-of-the-art RGB-only highlight removal frameworks, supporting effective generalization across scenes and surface types, notably in both natural and surgical imagery (Rota et al., 10 Dec 2025).

1. Construction of Synthetic Paired Data

At each training iteration, the system begins with a single unlabeled RGB image $\mathbf{I}^\star$. The primary objective is to create synthetic training pairs:

• The original image as a “diffuse” target, $\mathbf{I}^\star$,
• A “specularly augmented” image, $\hat{\mathbf{I}} = \mathbf{I}^\star +$ synthetic highlights, and
• A soft highlight mask $H$.

The process simulates the emergence of specularities by rendering physically-plausible highlights atop the diffuse input. This enables an RGB-only neural network to learn (i) to predict $H$ (the highlight segmentation) and (ii) to inpaint highlighted regions for recovery of the underlying diffuse appearance—all without explicit real-world paired data (Rota et al., 10 Dec 2025).

2. Physically-Based Highlight Rendering: Blinn–Phong with Fresnel Modulation

Virtual Highlight Synthesis employs a rendering pipeline rooted in the Blinn–Phong reflection model, further refined by a Schlick Fresnel term. This combination models specular behavior seen on non-Lambertian surfaces under point illumination. The following quantities are employed per pixel $p = (u, v, 1)^\top$:

• Camera intrinsics $K$,
• Monocularly estimated depth $D(p)$ and normal $n(p)$,
• The 3D point $X = D(p)\,K^{-1}p$,
• Light position $\mathbf{I}^\star$0,
• View ($\mathbf{I}^\star$1), light ($\mathbf{I}^\star$2), and half-vector ($\mathbf{I}^\star$3),
• Fresnel reflectance at normal incidence $\mathbf{I}^\star$4,
• Blinn–Phong exponent $\mathbf{I}^\star$5,
• Global specular scale $\mathbf{I}^\star$6.

The per-pixel Fresnel term is defined as:

$\mathbf{I}^\star$7

and the Blinn–Phong specular highlight is rendered as:

$\mathbf{I}^\star$8

Here, $\mathbf{I}^\star$9 expresses the synthetic highlight intensity.

The synthesis assumes: a single sampled point light per image, geometry estimated monocularly (accepting inherent error), and no interreflections or spatially varying BRDFs beyond the shininess $\hat{\mathbf{I}} = \mathbf{I}^\star +$0 (Rota et al., 10 Dec 2025).

3. Monocular Geometry Estimation

The pipeline relies on monocular geometric inference for accurate placement and shaping of synthetic highlights. A pretrained monocular network (e.g., MoGe-2) extracts both depth $\hat{\mathbf{I}} = \mathbf{I}^\star +$1 and normals $\hat{\mathbf{I}} = \mathbf{I}^\star +$2 for each pixel of $\hat{\mathbf{I}} = \mathbf{I}^\star +$3:

$\hat{\mathbf{I}} = \mathbf{I}^\star +$4

The 3D correspondence $\hat{\mathbf{I}} = \mathbf{I}^\star +$5 forms the basis for subsequent highlight rendering. The effectiveness of this stage is governed by the monocular estimator's accuracy, but the methodology is robust to realistic estimation errors when paired with randomized lighting (Rota et al., 10 Dec 2025).

4. Randomization of Point Light Sources

To improve diversity and prevent overfitting to specific highlight geometries, the illumination source $\hat{\mathbf{I}} = \mathbf{I}^\star +$6 is uniformly sampled within a bounded 3D volume in front of the camera at each training step. Once $\hat{\mathbf{I}} = \mathbf{I}^\star +$7 is selected, all highlight geometries and intensities for the image are computed accordingly. This randomization leads to varied highlight locations, shapes, and thresholds, thereby enhancing training robustness and model generalization (Rota et al., 10 Dec 2025).

5. Synthesis and Blending of Highlights with Diffuse Images

Synthetic specularities $\hat{\mathbf{I}} = \mathbf{I}^\star +$8 are compositively blended as additive masks into the diffuse image. The final “highlighted” image is computed by:

$\hat{\mathbf{I}} = \mathbf{I}^\star +$9

where $H$0 is the all-ones vector in RGB space, and brightening is modulated toward saturation depending on $H$1. This operation produces pseudo-paired triplets $H$2 suitable for fully-supervised training (Rota et al., 10 Dec 2025).

6. Network Supervision and Loss Design

The training regime leverages these synthetic pairs for end-to-end supervision of both highlight mask prediction and image inpainting:

• Highlight Mask Loss: The predicted mask $H$3 is supervised against $H$4 by a blend of soft-Dice, $H$5, and total variation (TV) regularization losses.
• Token-Space Inpainting: Feature tokens (from a frozen DINOv3 backbone) of highlighted regions are masked and reconstructed with a ViT-based inpainter. Supervision targets only patches with reliable (non-clipped) ground-truth features, using a hybrid $H$6–cosine loss.
• Decoder Pretraining and Fused Losses: Before highlight inpainting, the RGB decoder is pretrained as an autoencoder on DINO features, mixing $H$7 and structural similarity (SSIM) loss.
• Fine-tuning penalties: Subsequent fine-tuning introduces seam losses (for inpainted region boundaries), a Charbonnier penalty on oversaturated pixels, and a reconstruction loss outside original highlight regions.

All losses are weighted and summed to drive the network $H$8 to predict highlight maps and reconstruct highlight-free appearances from a single RGB input (Rota et al., 10 Dec 2025).

7. Empirical Performance and Impact

On datasets with ground-truth diffuse-specular pairs (PSD, SHIQ, SSHR), the approach achieves state-of-the-art SSIM and low masked MSE, on par with prior RGB-only and polarimetric methods. On unpaired (“no-GT”) benchmarks (CroMo, HouseCat6D, SCRREAM, Cholec80, SCARED, StereoMIS), Virtual Highlight Synthesis yields the lowest Luminance Suppression Ratio (LSR) and competitive NIQE scores, demonstrating effective highlight removal without global image darkening. Additionally, preprocessing with this pipeline improves downstream geometric consistency, reducing epipolar error and increasing inlier ratios for camera-pose estimation tasks across both natural and surgical data (Rota et al., 10 Dec 2025).

In summary, Virtual Highlight Synthesis orchestrates physically-based rendering on monocularly estimated geometry, randomized lighting, and Fresnel-modulated Blinn–Phong shading, which jointly enable the robust training of highlight removal networks directly on RGB imagery. This facilitates deployment in contexts where traditional paired supervision is infeasible while advancing the capacity for highlight removal, texture recovery, and downstream geometric reasoning in challenging, non-Lambertian scenarios (Rota et al., 10 Dec 2025).