Restoration-Gap Guidance (RGG)

• Restoration-Gap Guidance (RGG) is a framework that minimizes the discrepancy between degraded inputs and model-generated outputs across modalities.
• Key strategies include dual-branch architectures, learned gap predictors, and cross-modal signal injection to ensure both structural fidelity and perceptual detail.
• Empirical results show improvements in metrics like PSNR, FID, and RL returns, demonstrating RGG’s practical impact in image restoration and trajectory planning.

Restoration-Gap Guidance (RGG) designates a family of frameworks and algorithmic strategies that address the disconnect between information preserved in a degraded observation and the capacity of a generative or restoration model (often diffusion-based) to either retain or recover high-fidelity, semantically plausible structure. Across modalities—including images, trajectories, and cross-modal embeddings—RGG explicitly measures, monitors, or controls the “restoration gap”: the difference between what a degraded input can reliably provide and the extent to which a model’s generative prior “fills in” or potentially overrides this structure. This guidance mechanism is instantiated through architectural splits (e.g., dual-branch networks), learned predictors, or cross-modal signal injection, and is always governed by mathematically explicit objectives and algorithmic regularization schemes.

1. Theoretical Motivation and Formal Definitions

The restoration gap emerges when the conditional information available to a restoration or generative model underdetermines (fails to uniquely specify) the high-dimensional structure of the target output. For image super-resolution, the gap separates the low-frequency, downsampled latent (information preserved through heavy encoders) from high-frequency content (edges, texture, subtle structure) that is not recoverable by vanilla conditioning; for trajectory diffusion planners, the gap signals discrepancies between initial sampled plans and feasible, reward-maximizing or physically plausible behaviors.

Formally, the restoration gap ($g$) is defined for an input $x$ (image, trajectory, etc.) and a restoration operator $R_\theta$ as the expected deviation between the degraded input, sometimes after noise injection or projection, and the “restored” output following post-processing via the model or system:

$$\text{Restoration Gap} = \mathbb{E}_{\epsilon} \left[ \| x - R_\theta(\text{perturb}(x, \epsilon)) \| \right]$$

As in (Lee et al., 2023), for continuous-time trajectory planning under diffusion, the gap at noise level $\hat t$ can be estimated via Monte Carlo sampling of forward perturbations and integration of the reverse SDE.

In image recognition–aware restoration, the RGG objective is framed as a weighted combination of image-fidelity and downstream task (e.g. recognition) performance, balancing reconstruction loss with label-consistency across restored and clean-reference images (VidalMata et al., 2019):

$$\theta^* = \arg\min_\theta \left\{ \alpha L_\text{fid}(\theta) + \beta L_\text{recog}(\theta) \right\}$$

2. Algorithmic Strategies by Modality and Architecture

Image Restoration: GuideSR Dual-Branch RGG

GuideSR’s implementation of RGG (Arora et al., 1 May 2025) structurally disentangles the preservation of input-specific features from the injection of learned detail using a dual-branch architecture:

• Guidance Branch: Operates on the full-resolution input, extracting multi-scale structural features using stacked Full Resolution Blocks (FRBs), which are residual-in-residual units with channel attention. These features are refined via an Image Guidance Network (IGN) with guided attention to produce a residual image $R_2$—supervised to maintain input faithfulness.
• Diffusion Branch: Applies a pre-trained, LoRA-finetuned Stable Diffusion Turbo v2.1 prior to generate perceptually realistic detail. Single-step latent denoising is performed, with structural features from the Guidance Branch channel-concatenated into the UNet encoder at matching scales.
• Both output streams ($R_1$, $R_2$) are adversarially, perceptually, and pixel-wise supervised, with carefully selected balancing weights.

Trajectory Diffusion Planning

In diffusion-based offline control, RGG is realized by learning a gap predictor $G_\psi$ that estimates the restoration gap for noisy trajectory samples at any diffusion time $x$0, and by directly incorporating gradient information from this predictor (as a penalty) into the guided sampling SDE (Lee et al., 2023):

• Gap Predictor: Built atop frozen diffusion model encoders and trained to regression targets derived from Monte Carlo gap estimation.
• Guidance Integration: The reverse SDE for sampling is augmented with guidance from the reward gradient (as in reward-guided planning) and penalized by the gap-gradient, scaled by tunable hyperparameters.
• Attribution Map Regularization: Adds total-variation penalization to the temporal derivatives of gradient attribution maps of $x$1, mitigating the risk of adversarial or implausible gradient-driven excursions.

Cross-Modal and Text-Guided Image Restoration

RGG in cross-modal settings (e.g. (Lin et al., 2023)) leverages the ease of degradation removal in textual embedding space:

• Image-to-Text Mapping: Projects degraded image $x$2 into a finite set of pseudo-token embeddings via a CLIP image encoder plus a learned MLP.
• Textual Restoration Module: Purges degradation-predictive components from the token sequence using a second learned MLP.
• Guidance Generation: Restored tokens are used to synthesize a “clean” guidance image $x$3 using a frozen latent diffusion model (e.g., Stable Diffusion) conditioned on the text embedding.
• Coarse-to-Fine Injection: Multi-scale dynamic aggregation fuses $x$4 with backbone restoration network features, weighted and spatially aligned at each scale.

Blind Face Restoration with Dynamic RGG

DynFaceRestore (Do et al., 18 Jul 2025) advances RGG for unknown degradations by:

• Dynamic Blur-Level Mapping: Blindly degraded faces are mapped to a Gaussian blur regime via a trained estimator that outputs both a blur kernel and corresponding blurry image.
• Adaptive Diffusion Start: The starting timestep in the diffusion process is adaptively selected according to estimated blur, avoiding under- or over-diffusion.
• Dynamic Guidance Scaling: Local (pixel-wise or patch-wise) weighting of guidance strength via a CNN adjuster enables fine-grained control of fidelity-versus-hallucinated detail, ensuring sharp texture in relevant regions without sacrificing global structural faithfulness.

3. Loss Functions, Supervision, and Regularization

RGG approaches universally combine standard pixel-level or trajectory-level reconstruction losses with higher-order perceptual, adversarial, and (where applicable) downstream task-consistency losses:

• Image Losses: $x$5, $x$6 (feature-space), $x$7
• Multi-Branch Objectives: Weighted loss composition, often favoring the main generative output while preserving a contribution from structurally faithful side-branches (Arora et al., 1 May 2025).
• Cross-Modal Setting: Latent-diffusion reconstruction at both clean and degraded stages, $x$8 loss on backbone restoration output (Lin et al., 2023).
• Trajectory Setting: Supervision of the gap predictor to match Monte Carlo restoration gap estimates; the main sampling SDE is regularized by the gradient norm and total-variation on attribution maps (Lee et al., 2023).

4. Empirical Performance and Applications

RGG frameworks demonstrate consistent, often state-of-the-art, improvements across diverse benchmarks:

Task/Domain	Representative Gain Metrics	Reference
Image Super-Resolution	+1.39dB PSNR (DRealSR); FID–50.20; best MSE & LPIPS jointly	(Arora et al., 1 May 2025)
Blind Face Restoration	+0.4dB PSNR (CelebA-Test vs baseline); FID–0.25–1 pt; improved IDA	(Do et al., 18 Jul 2025)
All-in-One Restoration	+0.5dB PSNR, +0.003 SSIM (PromptIR backbone, BSD68/SOTS)	(Lin et al., 2023)
Diffusion Planning	+4.7–+9.4 RL avg return (Maze2D); +5–10 pts (Kuka stacking)	(Lee et al., 2023)

Notable themes in empirical analysis:

• Joint improvements in pixel-level fidelity and feature-space realism (balancing perception–distortion trade-off).
• Enhanced explainability in planning: attribution maps localize “gap-prone” transitions corresponding to physical infeasibility or semantic error (Lee et al., 2023).
• Robustness to multi-artifact and real-world degradations in large-scale benchmarks (VidalMata et al., 2019).

5. Key Hyperparameters and Implementation Patterns

Successful RGG implementations require careful design of architectural splits, guidance injection mechanisms, and calibration of loss scaling:

• GuideSR: $x$9 FRBs, $R_\theta$0 channels, LoRA ranks $R_\theta$1 (UNet) and $R_\theta$2 (VAE), $R_\theta$3-scaling in loss for branches (Arora et al., 1 May 2025).
• Gap Predictor: $R_\theta$4 MC samples, batch 32, $R_\theta$5 steps, $R_\theta$6 for high discriminative power (Lee et al., 2023).
• Cross-Modal: Tokens $R_\theta$7, $R_\theta$8; guidance scale $R_\theta$9 (Stable Diffusion) (Lin et al., 2023).
• DynFaceRestore: Blur-level estimator and dynamic scaling adjuster as shallow CNNs, starting timestep lookup per blur, patch-wise guidance weighting (Do et al., 18 Jul 2025).

6. Evaluation Metrics and Benchmarks

RGG methods are evaluated using composite suites of metrics reflecting both restoration and downstream performance:

• Reference-based: PSNR, SSIM, LPIPS, DISTS, FID (Arora et al., 1 May 2025, Do et al., 18 Jul 2025).
• Human Perceptual: Psychophysics-based Likert ratings (UG^2 challenge, (VidalMata et al., 2019)).
• Recognition-Driven: Top-1/Top-5 accuracy, multi-superclass inclusion rates (UG^2), ID-consistency for faces.
• Planning: RL average return, constraint satisfaction, artifact detection via restoration gap histograms (Lee et al., 2023).

RGG methods often achieve simultaneous improvements in both pixel-space and feature-space metrics, counter to the typical trade-off in pure perception-driven restoration.

7. Interpretability, Explainability, and Limitations

A salient feature of several RGG frameworks is algorithmic transparency:

• Attribution Map Regularization ensures that gradient-based guidance in trajectory (planning) tasks does not introduce adversarial transitions or exploit errors in the gap predictor (Lee et al., 2023).
• Ablation Studies confirm the necessity of architectural choices (e.g., multi-scale injection, proper conditioning modality) (Lin et al., 2023).
• Saliency and Attribution Visualization: High-gap regions correspond to semantically or physically implausible behavior (e.g., wall-crossing, object overlap), facilitating debugging and model improvement.

Limitations arise when gap predictors or cross-modal mappings are insufficiently expressive, or when the weighting between guidance and generative prior is poorly tuned. Some methods require substantial computational resources (e.g., frozen diffusion backbones, large MLPs for token mapping).

---

In summary, Restoration-Gap Guidance provides a principled, versatile set of mechanisms for minimizing the disconnect between degraded observations and semantically or physically consistent high-fidelity outputs in both vision and decision-making domains. Its explicit measurement and modulation of the information flow across restoration, generation, and guidance streams enable significant gains in both traditional and cross-modal restoration quality, interpretability, and downstream utility (Arora et al., 1 May 2025, Lee et al., 2023, Lin et al., 2023, Do et al., 18 Jul 2025, VidalMata et al., 2019).