Sign Language Video Synthesis via Loss-Guided Multi-Expert GANs

Abstract: This preliminary technical report presents a framework for sign language video synthesis using a loss-guided multi-expert Generative Adversarial Network (GAN) to enhance communication for individuals with hearing impairments. Three specialized discriminators -- global, hand, and head -- each guide a corresponding expert branch in the generator toward a distinct visual region, enabling implicit feature specialization without explicit diversity losses. To stabilize this multi-discriminator system, whose early-phase training otherwise exhibits chaotic dynamics, we introduce a United Loss consensus mechanism that regularizes each discriminator toward the ensemble average at a 10% weight. Each branch further adopts a dual-pathway convolutional-transformer design with learnable AdaptiveFeatureFusion, balancing the stability of convolutions against the detail of windowed self-attention. The generator is trained using an alternating three-mode schedule (discriminator, holistic generation, branch-specialized generation). On a custom 156GB dataset with a filtered test set that removes easy and repetitive samples, our 0.2B-parameter variant achieves 29.8 PSNR and the 1.3B-parameter variant achieves 30.7 PSNR, with inference VRAM footprints of 1.5 GB and 8 GB respectively, enabling deployment on consumer-grade hardware. Full ablation studies remain ongoing due to the 2-3 month training cycle on a single GPU. The system was showcased at the 2025 Hong Kong Frontier Technology Summit.

• The paper introduces a parallel U-Net generator with global, hand, and head expert branches supervised by specialized discriminators, achieving filtered-test PSNR scores of 29.8–30.7 across 0.2B–1.3B parameter models.
• United Loss stabilizes early three-discriminator training by encouraging adversarial-loss consensus, while learnable convolution–transformer fusion helps preserve fine details such as fingers, facial expressions, and clothing boundaries.
• The framework supports consumer-hardware deployment with 1.5–8 GB inference VRAM, but its benefits remain preliminary because controlled ablations, perceptual metrics, human evaluations, and action-accuracy tests are not yet reported.

Overview and Motivation

This technical report from Glassbox AI presents a preliminary framework for sign language video synthesis built on a multi-discriminator GAN (MD-GAN) in which three specialized discriminators—global, hand, and head—each supervise a corresponding expert branch of a parallel U-Net generator. The central premise is that sign language generation places disproportionate representational demands on small, high-detail regions (fingers, facial expressions), which a single global discriminator with a 448×448 receptive field cannot adequately supervise. By coupling region-specific discriminators to dedicated generator branches, the authors obtain implicit feature specialization without explicit diversity losses. A United Loss consensus mechanism regularizes each discriminator toward the ensemble average at a 10% weight to stabilize the otherwise chaotic early-phase dynamics of the three-optimizer system.

The work is explicitly positioned as preliminary: full ablation studies remain incomplete due to 2–3 month training cycles on a single consumer GPU, and evaluation is limited to PSNR on a filtered test set.

Architecture

Multi-Discriminator Design

The framework employs three discriminators with distinct operating regimes:

• Global discriminator: processes 448×448 images across five resolution levels (448 down to 28), using Haar wavelet transforms to expand input channels from 6 to 24 frequency subbands, MiniBatch Standard Deviation for mode-collapse prevention, and a PatchGAN-style output.
• Hand discriminator: operates on 112×112 patches localized via skeleton keypoints with center-crop alignment, across three scales.
• Head discriminator: processes facial regions defined by nose keypoints with a 96-pixel radius.

The Haar wavelet decomposition is intended to capture both structural (low-frequency) and textural (high-frequency) cues simultaneously, addressing the v3-era observation that fingers appeared unnaturally smooth under purely RGB supervision.

Generator: Dual-Pathway Multi-Parallel U-Net

The generator extends the Multi Parallel U-Net paradigm [ALJOWAIR2023104767] by activating all three branches (head, hand, global) simultaneously on identical 448×448 inputs, rather than using dynamic MoE routing. Each branch contains a two-stage dual-pathway block (Downsample_Vit):

1. Stage 1 fuses a channel-wise ConvTransformerBlock stream with a linear-projection/Attn_Conv2d stream via AdaptiveFeatureFusion (AFF).
2. Stage 2 fuses a pure convolutional stream with a Swin Transformer stream (W-MSA) via a second AFF module.

AFF computes $$\text{Fused} = \alpha \cdot \text{Stream}_1 + (1-\alpha) \cdot \text{Stream}_2$$ with learnable $\alpha$, replacing an earlier fixed scalar blend that proved insufficient. The authors attribute the design's effectiveness to opposing inductive biases under adversarial training: convolutions act as low-pass filters yielding stable but over-smoothed features, while windowed self-attention preserves high-frequency detail but is unstable at boundaries (clothing edges, finger contours). The learnable fusion restores balance per branch and depth level, and divergent discriminator-driven gradients prevent homogenization across branches.

The decoder mirrors this design with Upsample_Vit blocks, AdaIN-fused skip connections, keypoint-guided cross-attention injected at select layers (up_3, up_5, up_7), and a final single-channel Upsample_Transformer refinement block. A lightweight MappingNetwork (3-layer Transformer encoder, 8 heads) embeds 133-joint 3D skeleton coordinates into keypoints_info_f shared across decoder layers—a mechanism adopted after coordinate normalization caused train-test mismatch in v2, which dropout failed to fix but explicit keypoint injection resolved.

A Local-Global Merged Attention mechanism combines three spatial scales with fixed weights (0.85 local / 0.10 sub-local / 0.05 global), reflecting a stated design principle that local detail should dominate.

United Loss and Training Strategy

The core stabilization contribution is the United Loss consensus. Without it, the authors observe a runaway feedback loop in which the generator capitulates to whichever discriminator is momentarily easiest to satisfy, collapsing that discriminator's loss while the others' losses surge—effectively degenerating to single-discriminator training. The consensus term penalizes deviation asymmetrically: a discriminator whose adversarial loss has collapsed has its total loss dominated by the elevated ensemble average, pulling it back toward consensus, while high-loss discriminators remain dominated by their own terms and free to specialize.

Each discriminator's total loss blends its adversarial loss with the ensemble average at $\lambda_{\text{united}} = 0.1$, chosen empirically. The generator receives equal-weight feedback ($\lambda_g = \lambda_h = \lambda_f = 0.33$) computed via BCE against one-hot labels sampled as $r_{\text{real}} \sim \mathcal{U}(0.99, 1.0)$ and $r_{\text{fake}} \sim \mathcal{U}(0.0, 0.01)$.

Training follows a deterministic three-mode rotation, $\text{mode}(s) = \lfloor s/10 \rfloor \bmod 3$: (0) all discriminators update in parallel with the generator frozen; (1) holistic generator update through all parameters; (2) branch-specialized updates with independent per-branch optimizers. Samples without hands trigger only global updates. This fixed 1:1:1 schedule replaced an earlier cosine-scheduled phased-freezing design, which the authors report was less stable—an empirical claim offered without controlled evidence.

Notably, the authors characterize United Loss as a "training wheel": they observe that discriminator coordination emerges implicitly later in training, so the mechanism's influence matters primarily in the vulnerable early phase. This is a plausible hypothesis but remains unverified by controlled experiment on the current architecture.

Experiments and Results

Filtered Evaluation Protocol

Training uses a custom 156 GB dataset where each video follows a resting-pose → gesture → resting-pose structure. Resting-pose "easy samples" constitute roughly half of all frames and are trivially reconstructable; including them inflates PSNR beyond 36 without reflecting gesture quality. The reported metrics therefore use a filtered test set excluding easy samples entirely, targeting transition and gesture frames where generation is prone to collapse. This filtering decision is methodologically sound and makes the reported numbers more conservative than unfiltered alternatives.

Scale Results

Model	Params	PSNR (filtered)	Inference VRAM
Small	0.2B	29.8	1.5 GB
Medium	0.66B	30.4*	~5 GB
Large	1.3B	30.7	8 GB

*The 0.66B result comes from dataset v3 (lower contrast); retraining on v4 is expected to yield 30.1–30.2.

The monotonic PSNR improvement from 29.8 to 30.7 across a 6.5× parameter increase suggests effective capacity utilization rather than overfitting, though the medium-scale result's dataset mismatch weakens strict comparability. All variants train on a single RTX 4090 and infer within 1.5–8 GB VRAM, supporting deployment on consumer hardware—the paper's most practically significant claim.

Training Dynamics

PSNR progression over ~7M steps exhibits plateau-to-breakthrough phase transitions followed by terminal oscillation. The authors hypothesize these breakthroughs reflect United Loss consensus dynamics: when one branch discovers a beneficial representation, the consensus propagates improvements ensemble-wide, with cross-region improvements observed within 50,000–100,000 steps. Qualitative inspection confirms branch specialization despite identical inputs—sharper hands, more expressive faces, better body coherence—and training stability across scales without gradient penalty or spectral normalization. These observations are consistent with, but do not causally establish, the consensus mechanism.

Limitations and Open Questions

The paper is candid about several constraints:

• No controlled ablation: the contribution of United Loss, the dual-pathway design, alternating modes, and multi-scale attention have not been isolated on the v4 architecture. An ablation at $\lambda_{\text{united}} = 0$ and varied $\lambda_{\text{united}}$ is planned but blocked by the 2–3 month per-run cost. Causal attribution of stability to United Loss therefore rests on observational evidence plus an earlier, informal ablation during development.
• Single metric: only PSNR is reported. FID, LPIPS, action accuracy, and human evaluation are planned but absent, leaving the relationship between PSNR and perceptual quality—particularly for hands and faces—open.
• Scaling bottleneck: all branches activate at both training and inference, so adding experts incurs linear cost. The proposed remedy (label-routed expert selection with a pre-trained router) is future work, not a validated result.
• Dataset versioning: PSNR was rigorously recorded only on v4; earlier versions were evaluated subjectively, limiting longitudinal comparability.
• Fixed attention weights: the 0.85/0.10/0.05 Local-Global weighting is asserted as a design principle rather than derived or ablated.

Conclusion

This report describes a coherent engineering progression—from AdaIN-based style transfer (v1), through keypoint-guided generalization (v2) and local hand discrimination (v3), to a fully parallel three-expert architecture (v4)—culminating in filtered-test-set PSNR of 29.8–30.7 across 0.2B–1.3B parameter scales with consumer-grade inference footprints. The United Loss consensus mechanism and learnable dual-pathway fusion are the principal methodological contributions, each supported by mechanistic argument and qualitative observation rather than controlled experiment. The most pressing open questions are whether United Loss causally stabilizes three-discriminator training, how the architecture performs under perceptual and human evaluation, and whether label-routed expert selection can bound inference cost as expert count grows.