SpriteHand: GAN-Based Sprite Imputation

• SpriteHand is a method that frames sprite completion as a missing-data imputation task using the correlations among four directional poses.
• It employs a conditional GAN architecture with four encoder branches and a unified decoder to generate absent poses from available inputs.
• Experimental results show that SpriteHand outperforms Pix2Pix and StarGAN on FID and L1 metrics, with robust performance achieved through conservative input dropout.

SpriteHand is a GAN-based character sprite generator formulated as a missing-data imputation problem rather than a standard single-source image-to-image translation task. It operates on character sprite sheets decomposed into four pose domains—back, left, front, and right—and predicts a missing pose from the remaining available poses. In this formulation, sprite completion is treated as inference over a correlated multi-view representation of a character: if one directional sprite is absent or needs updating, the others provide the evidence used to synthesize it. The method is presented as a CollaGAN-inspired conditional GAN with architectural and training modifications tailored to pixel-art data, and it is evaluated on a large four-direction sprite dataset with comparisons against Pix2Pix and StarGAN baselines (Coutinho et al., 2024).

1. Problem formulation

SpriteHand models a character sprite sheet as a set of four aligned directional images. Let the pose domains be $N=\{a,b,c,d\}$, one per direction. The model learns to generate a missing target pose $t$ from the source set $S=N-\{t\}$:

$\hat{x}_t = G(x_S,t), \quad t\in N,\; S=N-\{t\}.$

This framing differs from pairwise pose translation. Instead of learning a mapping from a single source direction to a single target direction, SpriteHand conditions on all available poses and imputes the absent one. In the intended use case, this matches sprite-sheet editing more closely: artists often maintain a coherent set of directional views, and when one view changes or is missing, the other views should be used jointly to infer the absent frame.

The paper notes that the method can also be interpreted as a multi-input, multi-domain image-to-image translation task. The imputation framing, however, emphasizes both the training protocol and the assumption that the four views are correlated manifestations of the same underlying character. This suggests a design bias toward pose consistency rather than unrestricted visual synthesis.

2. Data representation and architectural design

Each example is a character sprite sheet decomposed into four aligned pose images. The representation uses four domains—back, left, front, right—with image size $64 \times 64 \times 4$. The four channels arise from RGBA encoding, and the alpha channel is retained rather than discarded. Smaller sprites are transparency-padded to $64\times64$, and if alpha is missing, an alpha mask is generated from the character shape. The reported split is 85/15, with 12,074 training examples and 2,128 test examples, drawn from a dataset totaling 14,202 paired multi-direction examples. Data augmentation uses hue rotation (Coutinho et al., 2024).

The generator has four encoder branches, one branch per available source pose, followed by a single bottleneck where all encoded features are concatenated, and then a single decoder with skip connections. Each source image is paired with the target-domain label; the label is one-hot encoded, spatially broadcast, and concatenated to the image input so that every encoder branch receives both the source pose image and the target label. Architecturally, the generator uses four downsampling blocks per encoder branch, a merged bottleneck, and four upsampling blocks in the decoder. The output is the generated image $\hat{x}_t$.

A specific architectural modification relative to the original CollaGAN is the number of encoder branches: the original CollaGAN used eight encoder branches, whereas SpriteHand uses four, matching the four sprite directions. The generator is also made much larger, increasing from 6,565,712 parameters in the original CollaGAN generator to 104,887,616 parameters in the proposed generator. The stated reason is to improve generation quality.

The discriminator is a dual-head critic with an adversarial head $D_{adv}$ for real/fake prediction and a domain-classification head $D_{dmn}$ for pose/domain prediction. Its topology retains the original CollaGAN structure: six downsampling blocks, Leaky ReLU activations, dropout in the final block, and two parallel output convolutions, one linear output for real/fake and one softmax output for domain classification.

3. Objective functions and optimization

SpriteHand uses a composite training objective. The generator is optimized with five terms: regression loss, multi-cycle consistency loss, SSIM loss, adversarial loss, and domain classification loss. The discriminator is optimized with adversarial loss and domain classification loss (Coutinho et al., 2024).

The regression term directly compares the generated target to the true target:

$$\mathcal{L}_{reg}=\mathbb{E}_{x_t,x_S}[\lVert x_t-\hat{x}_t\rVert_1].$$

The multi-cycle consistency term enforces that the generated target contains sufficient information to reconstruct the source images:

$t$0

The SSIM term is applied to the cyclically generated outputs and is described as the same as in the original CollaGAN. Adversarial training uses least-squares GAN objectives. The discriminator objective is

$t$1

and the generator adversarial objective is

$t$2

Domain classification is imposed on both real and generated samples:

$t$3

$t$4

The full objectives are

$t$5

$t$6

The reported coefficients are $t$7, $t$8, $t$9, and $S=N-\{t\}$0. During training, a minibatch with a randomly chosen missing target pose is given to the generator, the missing slot is filled with a zero tensor, and a multi-cycle backward pass reconstructs the source images from the generated target and the remaining inputs. This cyclic structure is central to the imputation interpretation: the forward prediction is constrained not only to match the target but also to preserve information needed to recover the observed views.

Training uses Adam with $S=N-\{t\}$1 and $S=N-\{t\}$2. The learning rate starts at 0.0001 and linearly decays to zero in the second half of training. The model is trained for 240,000 generator update steps, with batch size 4 and approximately 80 epochs. Checkpoints are evaluated every 1,000 steps, and model selection uses the best test $S=N-\{t\}$3. Reported training time is about 1:20 h on a GeForce GTX 1050; inference is about 110 ms per generated batch and about 116 ms per image as reported for the generator.

4. Missing-input robustness and training modifications

A defining feature of SpriteHand is that it is explicitly evaluated under different levels of missingness. The model is designed for one missing pose, corresponding to three available source images, but the same trained system is also tested when only two or one source images are available. The paper names these regimes CollaGAN-3, CollaGAN-2, and CollaGAN-1, respectively (Coutinho et al., 2024).

To improve robustness to absent source poses, the training procedure uses input dropout, where some source domains are replaced by zero tensors. Four strategies are compared: no dropout, original CollaGAN dropout, a curriculum strategy, and a conservative dropout strategy. The conservative strategy is reported as best overall: 60% chance to keep all 3 source images, 30% chance to drop 1 image, and 10% chance to drop 2 images. The stated effect is especially strong in the low-input regime.

A second training modification concerns replacement in the backward pass. In the original CollaGAN, the generated forward target $S=N-\{t\}$4 replaced both the true target image and any dropped-out images. The paper reports better results when $S=N-\{t\}$5 replaces only the original target image, while dropped-out inputs remain zero tensors. The reported rationale is that this avoids contamination from other domains and reduces artifacts such as extra facial details appearing in the wrong pose.

The architecture ablation is cumulative and isolates the effect of the proposed changes. The sequence reported is original CollaGAN, then increased capacity, then the forward replacer modification, then conservative input dropout. Simply increasing capacity improved $S=N-\{t\}$6 but hurt FID; the forward-only replacer improved both metrics; conservative dropout provided the final gain. This pattern indicates that SpriteHand’s improvements do not arise from parameter count alone.

5. Empirical results

The main quantitative comparison uses Pix2Pix and StarGAN as baselines. Pix2Pix is implemented as 12 pairwise models covering all source-target direction pairs. StarGAN is implemented as a single multi-domain model and is modified for better sprite quality by conditioning the generator on both source and target labels and conditioning the discriminator on the source image as well. Unlike original StarGAN, it is trained with supervision for fairness (Coutinho et al., 2024).

With three source images available, the proposed method outperformed the baselines on both FID and $S=N-\{t\}$7. The reported average FID values are 4.091 for Pix2Pix, 2.288 for StarGAN, and 1.508 for CollaGAN-3. The reported average $S=N-\{t\}$8 values are 0.05273 for Pix2Pix, 0.06577 for StarGAN, and 0.04078 for CollaGAN-3.

Setting	Avg FID	Avg $S=N-\{t\}$9
Pix2Pix	4.091	0.05273
StarGAN	2.288	0.06577
CollaGAN-3	1.508	0.04078

Performance degrades monotonically as fewer source poses are available. The reported averages are FID 8.393 and $\hat{x}_t = G(x_S,t), \quad t\in N,\; S=N-\{t\}.$0 0.06449 for CollaGAN-1, FID 4.277 and $\hat{x}_t = G(x_S,t), \quad t\in N,\; S=N-\{t\}.$1 0.05035 for CollaGAN-2, and FID 1.508 and $\hat{x}_t = G(x_S,t), \quad t\in N,\; S=N-\{t\}.$2 0.04078 for CollaGAN-3. CollaGAN-2 already beats Pix2Pix and StarGAN in $\hat{x}_t = G(x_S,t), \quad t\in N,\; S=N-\{t\}.$3, whereas CollaGAN-1 remains feasible but with substantially reduced quality.

The input-dropout ablation reports the following averages over the 3-source, 2-source, and 1-source test scenarios: no dropout yields FID 18.847 and $\hat{x}_t = G(x_S,t), \quad t\in N,\; S=N-\{t\}.$4 0.08449; original dropout yields FID 6.636 and $\hat{x}_t = G(x_S,t), \quad t\in N,\; S=N-\{t\}.$5 0.05191; curriculum yields FID 10.566 and $\hat{x}_t = G(x_S,t), \quad t\in N,\; S=N-\{t\}.$6 0.07948; conservative dropout yields FID 4.726 and $\hat{x}_t = G(x_S,t), \quad t\in N,\; S=N-\{t\}.$7 0.05187. The architecture ablation reports original CollaGAN at FID 8.866 and $\hat{x}_t = G(x_S,t), \quad t\in N,\; S=N-\{t\}.$8 0.06069, increased capacity at FID 11.078 and $\hat{x}_t = G(x_S,t), \quad t\in N,\; S=N-\{t\}.$9 0.05666, forward replacer at FID 6.636 and $64 \times 64 \times 4$0 0.05191, and conservative input dropout at FID 4.726 and $64 \times 64 \times 4$1 0.05187.

Qualitatively, generated sprites are described as generally sharp and pose-consistent. Left/right generation is especially strong, often because it reduces to near-horizontal flipping. Back-facing generation is weaker, with some face details sometimes bleeding through. Even with only one or two source images, outputs remain plausible, though fidelity declines. A recurring issue is color palette drift: the model may generate many slight variants of similar colors instead of strictly preserving a small sprite palette. The paper notes that post-processing quantization, as used in prior work, could reduce this problem.

6. Relation to adjacent sprite-generation and sprite-sheet methods

SpriteHand occupies a distinct position within recent sprite-generation research. Earlier work on "Generating Pixel Art Character Sprites using GANs" formulates sprite generation as supervised front-facing to right-facing image-to-image translation with a Pix2Pix-style U-Net generator, PatchGAN discriminator, RGBA inputs, and adversarial plus $64 \times 64 \times 4$2 losses. That approach is explicitly pairwise and requires separate mappings such as front-to-right, whereas SpriteHand uses one generator/discriminator pair and conditions jointly on multiple available directions (Coutinho et al., 2022).

The contrast is sharper with the "Five-Dollar Model," which addresses a different problem class: text-to-sprite generation from sentence embeddings. In that setting, the model produces an $64 \times 64 \times 4$3 one-hot sprite image from a 384-dimensional sentence embedding encoded by multi-qa-MiniLM-L6-cos-v1 plus a small noise vector, using a lightweight feedforward-convolutional generator with residual blocks and categorical cross-entropy training. That line of work emphasizes semantic prompt alignment, caption augmentation, and compositional color or attribute editing in a tiny 100-sprite dataset; SpriteHand instead assumes aligned multi-view character sheets and focuses on pose completion within a much larger four-direction corpus (Merino et al., 2023).

SpriteHand is also adjacent to sprite-sheet analysis and rigging rather than only generation. APES, "Articulated Part Extraction from Sprite Sheets," starts from a small set of character poses and infers articulated parts through correspondence estimation, motion-based clustering, part selection, and ARAP reconstruction. APES is relevant to SpriteHand-style workflows because it addresses the downstream problem of turning multi-pose sprite sheets into articulated rigs; SpriteHand addresses pose completion, while APES addresses decomposition into head, torso, limbs, and related parts suitable for puppet creation (Xu et al., 2022).

A different neighboring line is gameplay-conditioned sprite modification. "Generating Gameplay-Relevant Art Assets with Transfer Learning" uses a CVAE to modify Pokémon sprites based on type-conditioning vectors, emphasizing gameplay relevance, HSV representations, and transfer learning from anime faces. That work concerns controllable visual alteration of an existing sprite under mechanic-specific conditioning, not multi-view imputation of a missing directional pose. This suggests that SpriteHand is best understood as a structural completion method for directional consistency rather than a general semantic sprite editor (Gonzalez et al., 2020).

7. Limitations, applications, and future directions

The reported limitations are specific. Quality degrades when too many source poses are missing. Some pose-specific artifacts remain, especially for back views. Palette consistency is not perfectly preserved. The model is tested only on four-direction sprite data, so more complex animation or pose spaces remain open (Coutinho et al., 2024).

The stated applications include automated sprite editing, assisting game artists in maintaining consistency across directions, propagating design changes across sprite sheets, procedural content generation for 2D games, and future extension to animation frames, richer pose spaces, or diffusion-based sprite completion. The central conclusion is that character sprite generation benefits from being cast as missing-data imputation: when all available poses are used jointly, the model can infer missing views more accurately than single-source translation baselines.

The paper also suggests future work on ReMIC and MM-GAN for more flexible multi-domain imputation, disentangled content/style architectures, and latent diffusion models for higher-quality, multi-modal sprite suggestions. A plausible implication is that SpriteHand’s contribution lies less in introducing a wholly new adversarial paradigm than in identifying a problem formulation—multi-view imputation—and a set of targeted architectural and training changes that fit the geometry and workflow constraints of directional pixel-art sprite sheets.