NijiGAN: Real-to-Anime GAN Framework

• NijiGAN is a generative adversarial network framework for real-to-anime image translation, integrating continuous-depth NeuralODEs with a CUT-based architecture.
• It employs pseudo-paired data in a semi-supervised contrastive learning paradigm to achieve competitive FID and MOS scores while halving the parameter count.
• The architecture unifies multiple loss functions and leverages high-quality anime references to minimize artifacts and enhance visual fidelity.

NijiGAN is a generative adversarial network (GAN) framework designed for high-fidelity, real-to-anime image translation. It integrates Neural Ordinary Differential Equations (NeuralODEs) into a contrastive semi-supervised learning paradigm, aiming to improve visual quality and computational efficiency compared to prior models such as Scenimefy and AnimeGAN. NijiGAN is distinguished by its continuous-depth modeling in the generator architecture, reliance on pseudo-paired data for semi-supervised training, and the unification of multiple contrastive and adversarial losses to address domain gaps in image stylization (Santoso et al., 2024).

1. Architectural Overview

NijiGAN leverages a modified contrastive unpaired translation (CUT) generator architecture originally found in Scenimefy. The model consists of three principal components:

• Encoder ($G_{enc}$): A stack of convolutional layers transforms a $256 \times 256$ RGB image $x$ (real or pseudo-paired) into a multi-scale latent representation $h(0)$.
• Continuous-depth bottleneck (Neural ODE block): Instead of discrete residual (ResNet) blocks, the bottleneck is parameterized by a vector field $f_\theta$ and models the latent evolution via the initial value problem

$$\frac{dh(t)}{dt} = f_\theta(t, h(t)), \quad h(0) = h_0, \quad t \in [0, T],$$

where $h(T)$ is computed using the Dormand–Prince ODE solver and subsequently forwarded to the decoder.

• Decoder ($G_{dec}$): Upsampling (deconvolutional) layers reconstruct the anime-style image from $h(T)$.

The adversarial setup deploys two discriminators: $D_p$ (conditionally discriminates supervised pairs of real/pseudo input and output) and $256 \times 256$0 (distinguishes unpaired real anime images from generated outputs in the unsupervised branch) (Santoso et al., 2024).

2. NeuralODE Integration and Parameterization

Traditional ResNet-based architectures apply discrete residual updates, formally,

$256 \times 256$1

In the limit $256 \times 256$2, this process yields a continuous dynamical system,

$256 \times 256$3

with solution

$256 \times 256$4

NijiGAN implements $256 \times 256$5 as a convolution–batch norm–activation module, optimized using the Dormand–Prince RK45 solver. Replacing $256 \times 256$6 discrete ResNet blocks with a single NeuralODE block of tunable depth $256 \times 256$7 substantially reduces parameter count and potentially increases the model’s effective capacity to propagate features, as demonstrated by parameter count (5.477M) nearly halving the Scenimefy baseline (11.378M) (Santoso et al., 2024).

3. Semi-Supervised Contrastive Learning

NijiGAN’s training objective merges supervised and unsupervised translation through a two-branch semi-supervised scheme:

• Supervised branch: Utilizes pseudo-paired data $256 \times 256$8 generated by Scenimefy, preventing reliance on artifacts from StyleGAN-produced low-quality pairs.

Key supervised losses: - Conditional GAN loss ($256 \times 256$9): A patch-based discriminative loss operating on channel-wise concatenated input-output pairs. - Patch-wise contrastive loss (StylePatchNCE): Multi-scale contrastive loss for style features between generated and reference patches. - Supervised aggregate loss:

$x$0

where $x$1.

• Unsupervised branch: Operates over unpaired LHQ images and anime references.
  • GAN loss ($x$2) using PatchGAN.
  • Semantic relation consistency (SRC) loss:

  $x$3

  enforcing invariance in internal feature similarity distributions. - Hard negative contrastive loss (hDCE): Penalizes the model for failing to differentiate true negatives in representation space.

The aggregate unsupervised loss is:

$x$4

with $x$5.

• Total objective: Weighted sum annealed over training epochs $x$6:

$x$7

4. Training Datasets and Optimization

Training utilizes the following datasets:

• Pseudo-paired set: 24,000 LHQ images and Scenimefy-generated anime translations, resized to $x$8.
• Unpaired real-world inputs: 10,000 LHQ images (real-world photos).
• Anime references: 4,821 frames from five Makoto Shinkai anime films, providing diverse, high-quality style exemplars.

Optimization details:

• Adam optimizer $x$9, 25 epochs, approximately 24,000 steps per epoch.
• Batch size is not specified, but is typically 1–4 for $h(0)$0 images in CUT-based networks.
• $h(0)$1 throughout.
• The initial 10 epochs anneal in supervised loss, beginning with $h(0)$2.

5. Empirical Results and Ablation Studies

Quantitative and qualitative evaluation demonstrates NijiGAN’s trade-offs in fidelity, parameter efficiency, and artifact suppression.

Model	FID (↓)	MOS (↑)	Parameters (M)
CartoonGAN	45.79	2.708	—
AnimeGAN	56.88	2.160	—
Scenimefy	60.32	2.232	11.378
NijiGAN	58.71	2.192	5.477

NijiGAN closes the gap with Scenimefy on FID (58.71 vs 60.32) and achieves comparable MOS (2.192 vs 2.232), while halving parameter count. Efficiency benchmarking shows equivalent peak GPU memory (9.694 GB for NijiGAN vs 9.718 GB for Scenimefy), slightly lower throughput (1.533 s vs 1.253 s per 5-run batch), and higher TFLOPS (0.771 vs 0.387).

Ablation studies reveal:

• Removal of NeuralODE or contrastive learning degrades perceptual quality, increasing blur and artifacts and worsening FID/MOS.
• VGG perceptual loss is integral for structure retention, but excessive weighting (e.g., $h(0)$3) over-smooths anime features.
• ResNet blocks reintroduce checkerboard artifacts and inflate parameter count.

6. Key Contributions and Dataset Release

NijiGAN's technical innovations and practical outcomes include:

• The integration of NeuralODEs into the CUT generator, reducing parameters by approximately half while achieving competitive or improved visual fidelity.
• The elimination of low-quality, StyleGAN-generated paired data, instead leveraging Scenimefy pseudo-pairs for supervised signals.
• Unified semi-supervised training incorporating patchwise contrastive, conditional GAN, semantic relation consistency, and hard negative losses, which collectively minimize domain shifts in unpaired translation.
• Empirical demonstration of reduced checkerboard artifacts, improved smoothness, and stable training across metrics.
• Introduction and public release of a new high-resolution anime scene dataset (4,821 Makoto Shinkai frames), expanding resources for future scene stylization research (Santoso et al., 2024).

7. Context and Impact

NijiGAN situates itself within the trajectory of generative scene translation, building on CUT-style contrastive learning and semi-supervised objectives while advancing architectural efficiency through continuous-depth modeling. Its release of a new anime reference dataset addresses a data scarcity in high-fidelity style transfer research. Its methodological structure—combining NeuralODEs, contrastive learning, and pseudo-pair supervision—establishes a reference point for subsequent models aiming to balance translation quality, parameter efficiency, and practical deployment constraints in real-to-anime image generation (Santoso et al., 2024).