Balanced Twin Perceptual Loss

• The paper introduces a balanced twin perceptual loss that fuses anime-specific ResNet features with photorealistic VGG signals to overcome standard loss limitations.
• It employs a two-stage training strategy with XDOG-based pseudo-ground-truth generation to accentuate hand-drawn lines and suppress color artifacts.
• Quantitative evaluations show improvements in NIQE, MANIQA, and CLIP-IQA scores, indicating enhanced structural fidelity and color stability in anime super-resolution.

Balanced twin perceptual loss is an objective function introduced for real-world anime super-resolution tasks designed to address the domain-specific challenges arising from the discrepancy between anime and photorealistic image statistics. Standard VGG-based perceptual loss, originally developed for natural images, can introduce color artifacts and fail to capture critical anime characteristics such as crisp hand-drawn lines. The balanced twin perceptual loss specifically combines high-level features from both an anime-domain network (ResNet50 pre-trained for anime image classification) and a photorealistic-domain network (VGG19 pre-trained on ImageNet) to guide generative models towards reconstructions that are both artifact-free and visually coherent in the anime domain (Wang et al., 2024).

1. Motivation and Background

Conventional perceptual loss functions, such as those based solely on VGG19 feature spaces, are suboptimal for anime imagery. Generative adversarial networks (GANs) fine-tuned with such losses tend to produce results with color instabilities and fail to maintain the unique structural qualities of hand-drawn lines in anime frames. The balanced twin perceptual loss was developed to mitigate two primary defects:

• Distorted and faint hand-drawn lines: Standard losses are insensitive to anime linework, leading to blurred or eroded edges.
• Unwanted color artifacts: VGG-based supervision may induce color hallucinations due to the mismatch between anime color distributions and those learned from natural images.

This approach directly addresses the requirements of real-world anime SR pipelines, where artifact suppression and line clarity are critical for production-grade outputs (Wang et al., 2024).

2. Mathematical Formulation

The balanced twin perceptual loss is formulated as the sum of two perceptual losses, each computed from different feature spaces.

Let $\phi_j(\cdot)$ denote the $j$-th intermediate feature map of a ResNet50 model pre-trained on anime images (Danbooru-classification), and $\psi_j(\cdot)$ denote the analogous feature map of a VGG19 model pre-trained on ImageNet. For reference, $I_{\text{SR}}$ is the super-resolved (predicted) image, and $I_{\text{pseudo}}$ is the pseudo-ground-truth image with enhanced linework.

The per-domain perceptual losses are defined as: $$L_{\text{ResNet}} = \sum_{j=1}^5 w^R_j \frac{1}{C_j H_j W_j} \lVert \phi_j(I_{\text{pseudo}}) - \phi_j(I_{\text{SR}}) \rVert_1$$

$$L_{\text{VGG}} = \sum_{j=1}^5 w^V_j \frac{1}{C_j H_j W_j} \lVert \psi_j(I_{\text{pseudo}}) - \psi_j(I_{\text{SR}}) \rVert_1$$

where $C_j$, $H_j$, and $W_j$ are the channel, height, and width dimensions of the feature maps at layer $j$, and $w^R_j$, $w^V_j$ are empirically determined weights.

The total loss is: $$L_{\text{total}} = \alpha L_1 + \beta (L_{\text{ResNet}} + \delta L_{\text{VGG}}) + \gamma L_{\text{adv}}$$ with $\alpha=1.0$, $\beta=0.5$, $\gamma=0.2$, $\delta=1.0$. Here, $L_1$ is pixelwise loss, and $L_{\text{adv}}$ is the adversarial loss from the GAN discriminator (Wang et al., 2024).

Empirical Weighting

The layer weights are set as:

• $w^R = \{0.1, 20, 25, 1, 1\}$
• $w^V = \{0.1, 1, 1, 1, 1\}$

3. Implementation Context and Pseudo-Ground-Truth Generation

Balanced twin perceptual loss is applied within a two-stage training scheme:

1. Stage 1: $L_1$ reconstruction only (“warmstart”; 300K iterations)
2. Stage 2: Add twin perceptual and adversarial objectives for another 300K iterations

The pseudo-ground-truth ($I_{\text{pseudo}}$) is generated by accentuating hand-drawn lines via iterative unsharp mask enhancement and binary edge extraction (XDOG), allowing the network to explicitly learn edge-localized fidelity (Wang et al., 2024).

This pipeline is essential for overcoming the weak or degraded linework present in anime video streams and aligns the training signal to the visual priorities of anime production.

4. Advantages Over Single-Domain Perceptual Loss

Single-domain perceptual losses (e.g., VGG-oriented) tend to strongly bias reconstruction toward their trained feature distributions, neglecting aspects that are not adequately represented in the training data, such as anime-specific linework and flat-shaded regions. Balanced twin perceptual loss effectively fuses:

• Anime-domain guidance (ResNet): Sensitive to semantic concepts and structural priors unique to the anime domain (lines, cell shading, stylized forms).
• Photorealistic guidance (VGG): Provides color stability and prevents degenerate solutions and global chromaticity errors.

This dual signal eliminates GAN-induced color speckles and supresses undesirable compression or generative artifacts, while maintaining clarity on hand-crafted lines—critical for both perceptual and objective quality (Wang et al., 2024).

5. Quantitative and Qualitative Impact

Ablation studies show significant improvements when using the balanced twin perceptual loss:

• NIQE (↓): Reduced from 7.351 (no twin perceptual, plain SR) to 6.719 (full pipeline).
• MANIQA (↑): Increased to 0.514, highest among compared methods.
• CLIP-IQA (↑): 0.711 versus 0.567 for the next best (VQD-SR) approach.

The methodology achieves superior line clarity (no oversharpened halos), blocks/ringing suppression, and preserved chromatic uniformity. All results are reported on the AVC-RealLQ test set, with training performed on the API dataset (3,740 images), representing only 13.3% of the data volume used by competing AnimeSR methods (Wang et al., 2024).

6. Practical Considerations and Limitations

Balanced twin perceptual loss depends upon the availability of discriminative anime-domain features and effective pseudo-ground-truth edge maps. The selection of weightings ($w_j^R$, $w_j^V$, $\alpha$, $\beta$, $\gamma$, $\delta$) is empirical and may require domain-specific tuning for datasets with different characteristics.

Additionally, applicability is mainly evidenced in anime SR tasks; its transferability to other highly stylized or out-of-domain settings is not established in the current literature.

7. Relation to Broader Research and Future Directions

Balanced twin perceptual loss represents an overview of domain transfer and content-aware perceptual supervision, closely related to works on domain-adapted GAN loss engineering and dataset-specific network pre-training. Its integration into compact SR pipelines fits the emerging trend of leveraging production-informed priors for real-world media restoration and synthesis.

Further lines of inquiry include extending the approach to video SR with temporal perceptual constraints, automating domain weighting via adaptive loss rebalancing, and exploring perceptual losses conditioned on higher-level structure (e.g., pose, lineart, semantic segmentation) in other animation or fine-art domains (Wang et al., 2024).