TGRN: Texture-Guided Restoration for BFR

• The paper introduces TGRN, a neural architecture that decomposes, aligns, and fuses identity-preserving and texture-rich features using a U-Net backbone with texture attention.
• TGRN employs dynamic fusion blocks with MLP-based channel weighting and multi-loss optimization to balance photorealism and identity fidelity under severe degradations.
• The integration of TGRN in CodeFormer++ achieves state-of-the-art blind face restoration, as evidenced by improved metrics such as PSNR, SSIM, and FID on diverse datasets.

Texture-Guided Restoration Network (TGRN) is a neural architecture central to CodeFormer++ for blind face restoration (BFR), which targets the decomposition, alignment, and fusion of identity-preserving and texture-rich features. TGRN dynamically transfers semantically aligned generative textures onto an identity-faithful restoration backbone through a lightweight U-Net–based encoder–decoder, augmented by a texture attention mechanism and dynamic fusion modules. It is designed to address the visual-quality/identity-fidelity trade-off seen in generative BFR methods by adaptively merging identity-consistent but texture-poor outputs with texture-strong but identity-drifting priors, enabling high-fidelity face recovery under severe and unknown image degradations (Reddem et al., 6 Oct 2025).

1. Architectural Structure and Feature Fusion

TGRN operates on two main inputs: the identity-preserving restoration $I_F$ (the output of CodeFormer's controllable feature transform with full identity weight) and the aligned generative prior $I_{\text{warp}}$ (the generative output warped via a dedicated deformable alignment module). Its core structure is a 3-level U-Net encoder–decoder, with each encoder stage augmented by a Texture Attention Module (TAM) and a dynamic fusion block.

At each encoder level $i$, the U-Net processes $I_F$, producing spatial-channel features $Z^i_e \in \mathbb{R}^{m\times n \times d}$. In parallel, $I_{\text{warp}}$ passes through a CNN, residual blocks, and adaptive pooling to yield texture features $Z^i_t \in \mathbb{R}^{m\times n \times d}$.

The dynamic fusion block computes spatially averaged descriptors: $$v^i_e = \frac{1}{mn}\sum_{s,t} Z^i_e(s,t), \qquad v^i_t = \frac{1}{mn}\sum_{s,t} Z^i_t(s,t)$$ These are concatenated and passed through a three-layer MLP to predict per-channel fusion weights $(w^i_e, w^i_t)\in\mathbb{R}^d$, which then mix the encoder ($Z^i_e$) and texture ($Z^i_t$) features: $Z^i_m = w^i_e \odot Z^i_e + w^i_t \odot Z^i_t$ The fused features $\{Z^i_m\}_{i=1}^3$ are propagated through the decoder to reconstruct the final restoration $I_{\mathrm{out}}$.

2. Mathematical Formulation

TGRN’s process and adaptive fusion are encapsulated by three principal equations:

1. Global average pooling for dynamic fusion:

$$v^i_e = \frac{1}{m\,n} \textstyle\sum_{s,t} Z^i_e(s,t), \quad v^i_t = \frac{1}{m\,n} \sum_{s,t} Z^i_t(s,t)$$

2. MLP-based channel weighting:

$$[\,w^i_e,\;w^i_t\,] = \mathrm{MLP}([v^i_e, v^i_t]), \quad w^i_e, w^i_t \in \mathbb R^d$$

3. Feature fusion by adaptive channel mixing:

$Z^i_m = w^i_e \odot Z^i_e + w^i_t \odot Z^i_t$

Here, $\odot$ denotes elementwise multiplication broadcast per channel, and all computation is performed at each encoder depth.

3. Training Losses and Optimization

TGRN is optimized using a composite loss that ensures photorealism, identity preservation, and robust fusion of semantic texture. The overall objective is: $$L_{\text{total}} = \lambda_{1} L_1 + \lambda_{\text{adv}} L_{\text{adv}} + \lambda_{\text{id}} L_{\text{id}} + L_{\text{triplet}}$$ where the weights $$(\lambda_1, \lambda_{\text{adv}}, \lambda_{\text{id}}) = (0.1, 0.1, 10)$$. The components are:

• $L_1$ loss (reconstruction):

$$L_1 = \left\| I_{\text{HQ}} - I_{\text{out}} \right\|_1$$

• Adversarial loss (softplus):

$$L_{\text{adv}} = -\mathbb{E}_{I_{\text{out}}} [\mathrm{softplus}(D(I_{\text{out}}))]$$

where $D$ is a learned discriminator.

• Identity loss (ArcFace backbone):

$$L_{\text{id}} = \| \eta(I_{\text{HQ}}) - \eta(I_{\text{out}}) \|_1$$

• Cosine-triplet loss (deep metric learning):
  • Let $I_{AP}=I_F \odot M + I_{\text{warp}} \odot (1-M)$ (with $M$ a mask of facial regions), and
  • $f_p, f_a, f_n$ the L2-normalized VGG features for $I_{AP}$, $I_{\text{out}}$, and $I_F$, respectively.
  • Triplet loss:

  $$L_{\mathrm{triplet}} = -\lambda_{\mathrm{triplet}} \log\left(\frac{e^{\cos\theta^+}}{e^{\cos\theta^+}+e^{\cos\theta^-}}\right),\quad \cos\theta^+ = f_p^\top f_a,\;\; \cos\theta^- = f_n^\top f_a$$

  with $\lambda_{\mathrm{triplet}} = 1$.

The combination of photometric, adversarial, identity, and novel triplet loss terms is crucial in balancing detail realism and identity fidelity during training (Reddem et al., 6 Oct 2025).

4. Pipeline Integration and Context in CodeFormer++

Within the CodeFormer++ pipeline, TGRN occupies a final fusion role after controllable feature extraction and deformable alignment:

1. The input $I_{\text{LQ}}$ is processed by CodeFormer’s controllable feature transform to obtain:

   • $I_F$ (identity-preserving, low-texture) with feature-weight $w=1$,
   • $I_G$ (texture-rich, permits identity drift) with $w=0$.
2. The Deformable Alignment Module (DAM) computes a flow $\phi = R_\theta(I_F, I_G)$ that warps $I_G$ to $I_{\text{warp}}$, semantically aligning generative texture with the structure of $I_F$.
3. TGRN receives $(I_F, I_{\text{warp}})$ and deploys the U-Net backbone, texture attention, and dynamic fusion, yielding $I_{\text{out}}$.
4. Triplet metric learning, with anchor $I_F$ (hard negative) and $I_{AP}$ (mask-wise blend as positive), enforces that $I_{\text{out}}$ absorbs texture selectively while maintaining identity congruence.

This stepwise arrangement enables CodeFormer++ to outperform approaches focusing solely on either identity or detail, by explicitly modeling and injecting generative priors into an aligned, identity-anchored backbone (Reddem et al., 6 Oct 2025).

5. Empirical Impact and Ablation

Quantitative experiments on both synthetic and real-world face datasets confirm the effectiveness of TGRN as a texture-injection and fusion module. Performance metrics include PSNR, SSIM, NIQE, LPIPS, FID, and LMD.

Table 1. CelebA-Test (synthetic)

Methods	PSNR ↑	SSIM ↑	NIQE ↓	LPIPS ↓	FID ↓	LMD ↓
CodeFormer	22.18	0.610	4.520	0.299	60.62	5.38
VQFR	24.14	0.636	3.693	0.351	41.28	9.13
Ours	24.96	0.697	4.052	0.341	38.13	5.41

Table 2. Real-world datasets (FID/NIQE)

Dataset	GPEN FID	GFPGAN FID	CodeFormer FID	Ours FID	Ours NIQE
LFW-Test	57.58	49.96	52.02	45.63	3.518
WebPhoto	81.77	87.35	78.87	72.91	3.822
WIDER-Test	46.99	39.73	39.06	35.21	3.482

Qualitatively, TGRN injects realistic mesoscopic skin textures (e.g., pores, fine wrinkles) into the CF-ID backbone, while preserving global face structure and identity, avoiding the excessive drift observed in purely generative priors.

Ablation studies demonstrate:

• DAM alone improves identity consistency but introduces artifacts,
• TGRN with only $L_1$/$L_{\text{id}}$/$L_{\text{adv}}$ reduces artifacts but lacks rich detail,
• Only the combined TGRN with novel triplet loss achieves optimal perceptual quality/identity fidelity trade-off.

6. Relationship to Other Texture-Guided Restoration Paradigms

While TGRN is specialized for BFR and operates in tandem with deformable registration and metric learning, its guiding principle—explicit, adaptive texture fusion—relates to the broader class of texture map–guided restoration networks. For instance, Fu et al. (Fu et al., 2020) propose the use of weak texture information maps derived from Sobel edge differences to aid super-resolution networks. The maps are predicted using a dedicated auxiliary network and injected via a fusion module to the main RCAN super-resolver, resulting in enhanced recovery of faint textures in SISR contexts.

A plausible implication is that while the weak-texture methods operate at the level of gradient maps and are relatively generic, TGRN’s fusion paradigm operates semantically (via deep, aligned features) and is tightly coupled with deep metric learning for identity-sensitive fusion—characteristics demanded by the BFR setting.

7. Significance and Prospects

TGRN provides a parameter-efficient solution to the dual requirements of detailed visual realism and faithful identity restoration. Through dynamic channelwise fusion and selective metric supervision, it establishes a new state-of-the-art in BFR tasks as evidenced by both quantitative metrics and expert-assessed perceptual quality.

An open direction suggested by the current design is the adaptation of TGRN-style dynamic fusion to other restoration settings, possibly substituting manually engineered texture maps (e.g., Sobel gradients) with deep feature–aligned texture modules for tasks such as general image restoration or SISR, extending the scope demonstrated by (Fu et al., 2020). Further research may explore more lightweight fusion architectures, self-supervised alignment, or hybridization of learned and hand-crafted texture priors.

References:

• "CodeFormer++: Blind Face Restoration Using Deformable Registration and Deep Metric Learning" (Reddem et al., 6 Oct 2025)
• "Weak Texture Information Map Guided Image Super-resolution with Deep Residual Networks" (Fu et al., 2020)