Sign-to-Speech Prosody Transfer via Sign Reconstruction-based GAN

Abstract: Deep learning models have improved sign language-to-text translation and made it easier for non-signers to understand signed messages. When the goal is spoken communication, a naive approach is to convert signed messages into text and then synthesize speech via Text-to-Speech (TTS). However, this two-stage pipeline inevitably treat text as a bottleneck representation, causing the loss of rich non-verbal information originally conveyed in the signing. To address this limitation, we propose a novel task, \emph{Sign-to-Speech Prosody Transfer}, which aims to capture the global prosodic nuances expressed in sign language and directly integrate them into synthesized speech. A major challenge is that aligning sign and speech requires expert knowledge, making annotation extremely costly and preventing the construction of large parallel corpora. To overcome this, we introduce \emph{SignRecGAN}, a scalable training framework that leverages unimodal datasets without cross-modal annotations through adversarial learning and reconstruction losses. Furthermore, we propose \emph{S2PFormer}, a new model architecture that preserves the expressive power of existing TTS models while enabling the injection of sign-derived prosody into the synthesized speech. Extensive experiments demonstrate that the proposed method can synthesize speech that faithfully reflects the emotional content of sign language, thereby opening new possibilities for more natural sign language communication. Our code will be available upon acceptance.

• The paper presents a novel sign-to-speech prosody transfer approach using a sign reconstruction-based GAN that bypasses text mediation.
• It introduces S2PFormer, which integrates visual feature extraction, adaptive prosody mixing, and explicit prosody estimation for natural speech synthesis.
• Objective and subjective evaluations confirm enhanced expressiveness and accurate reflection of sign intent without compromising speech intelligibility.

Sign-to-Speech Prosody Transfer via Sign Reconstruction-based GAN

Introduction and Motivation

The paper "Sign-to-Speech Prosody Transfer via Sign Reconstruction-based GAN" (2604.10413) introduces a new paradigm for translating prosodic nuances from sign language directly into synthesized speech. Traditional pipelines for sign-to-speech treat sign-to-text and text-to-speech as a cascade, with the intermediate textual representation serving as a severe information bottleneck that discards expressive prosodic cues such as emphasis, emotion, and tension inherently represented in the kinematics and facial expressions of signers. This work proposes an end-to-end multimodal prosody transfer system that seeks to inject these subtle and critical nuances into spoken output, avoiding the impoverishment inherent to text-mediated pipelines.

Figure 1: The proposed model (right) injects sign-emphasized expressions into speech, which are absent in text-based two-stage pipelines (middle).

Task Formulation: Sign-to-Speech Prosody Transfer

The authors rigorously define the novel task of sign-to-speech prosody transfer as the direct conditioning of synthesized spoken output on the prosodic information embedded in sign language videos. Unlike existing cross-modal prosody transfer work, which operates almost exclusively within the domain of spoken language and relies on abundant parallel corpora, the alignment of sign language motion and speech prosody demands expensive and scarce annotations. The transfer task is thus situated at the level of global, utterance-driven prosody. The output speech must reflect the signer's intent, including affective coloration, emphasis, and temporal phrasing, without textual mediation.

Methodology: SignRecGAN and S2PFormer

To solve the annotation bottleneck, the authors present SignRecGAN: a GAN-based framework that leverages unpaired unimodal datasets — large-scale sign language corpora and high-quality speech datasets — with no requirement for explicit cross-modal alignment. The key to the approach is to use adversarial training and cycle-consistent reconstruction objectives to force the synthesized speech to faithfully encode the global prosodic structure of the original sign sequence.

Figure 2: The SignRecGAN learning framework connects sign-to-speech prosody transfer through adversarial and reconstruction losses across unimodal domains.

S2PFormer Architecture

SignRecGAN is instantiated with S2PFormer, an augmented FastSpeech2-based transformer. S2PFormer incorporates visual feature extraction and cross-modal attention modules:

Figure 3: S2PFormer intake pipeline, leveraging a visual backbone for sign features, AdaPM for adaptive prosody fusion, and explicit prosody estimation.

• Visual Backbone: Utilizes GloFE, combining CTR-GCN and MS-TCN, to extract abstract temporal and relational features from keypoint sequences of hands and face.
• Adaptive Prosody Mixer (AdaPM): Mixture-of-Experts module adaptively blends variance predictors from sign and text domains.

Figure 4: Adaptive Prosody Mixer (AdaPM) adaptively fuses textual and sign-driven prosody representations.

• Prosody Estimator: Cross-modal mapping from speech pitch and energy contours to sign-derived motion histograms, facilitating explicit prosody reconstruction.

Losses and Training Objectives

• SignRec Loss: Forces the speech prosody decoder to reconstruct sign motion statistics (velocity/acceleration of hands and face) from synthesized mel-spectrograms, emphasizing the preservation of non-verbal cues.

Figure 5: Distributional structure of sign prosody labels, computed from hand and face motion histograms.

• ProMo Loss: Regularizes correspondence between speech pitch/energy and the magnitude/speed of upper-body and facial motion, enforcing prior knowledge from the speech-sign prosody literature.
• GAN Loss: LSGAN-based adversarial loss enforces speech realism.
• Intonation and Speaker Regularization: Constrains generated pitch/energy statistics to be within speaker-specific norms to avoid speaker drift and intonationally flat output.

Experimental Results and Analysis

Training is performed with OpenASL (sign-text pairs) and VCTK (clean, multi-speaker speech). Extensive ablation underscores the necessity of combining adversarial, reconstruction, and cross-modal regularization terms for optimal expressiveness without degradation in speech quality.

Objective Evaluation: The system yields pitch and energy standard deviations (representing expressiveness) substantially closer to real speech than the text-mediated baseline, while maintaining comparable word error rate (WER) and UTMOS (naturalness) scores relative to the best two-stage pipeline.

Subjective Evaluation: CMOS user studies (n=17) confirm that synthesized speech more accurately reflects sign-intended prosody, especially in utterances with substantial expressive content. No statistically significant degradation is observed for neutral samples, indicating robustness to overfitting expressiveness.

Figure 6: Example of sign input video (left) and corresponding prosody-infused speech output (right).

Prominence Analysis: Detailed analyses using the Wavelet Prosody Toolkit reveal the model's ability to emphasize contextually appropriate words in speech, reflecting nuanced motion cues in sign input.

Figure 7: Prominence analysis demonstrates systematically increased emphasis on semantically salient words.

Implications and Future Directions

This work establishes that robust global prosody transfer is feasible in the absence of parallel data, provided adversarial and cross-modal signals jointly inform the mapping. The results suggest the possibility of highly expressive, accessible spoken communication driven by visual sign input, extending existing human-computer interaction systems for the Deaf community.

On the theoretical front, the integration of adversarial and cycle-consistent objectives across coupled but unaligned multimodal time series is demonstrated to be effective for implicit prosody mapping. The modular design of S2PFormer permits straightforward integration with future advances in TTS and visual backbone modeling. The explicit modeling of non-verbal signals also invites further investigation into fine-grained, temporally precise prosody transfer, potentially in the context of dense gesture-speech alignment corpora.

Conclusion

The proposed SignRecGAN and S2PFormer deliver an end-to-end pathway for transferring prosodic intent from sign language to speech, overcoming the alignment and annotation bottleneck through scalable unpaired training. The resulting speech output is measurably more expressive, reflecting both global and fine-grained aspects of sign-driven prosody. The approach sets the stage for more naturalistic and inclusive sign language interfaces and motivates continued exploration at the intersection of multimodal learning, prosody modeling, and accessible AI systems.