Phonetic Synchronization in Speech Tech

• Phonetic Synchronization is the alignment of phonetic and articulatory details across audio, video, and text to ensure natural speech and lip motion coherence.
• Techniques combine temporal alignment, vowel matching, and context-aware modeling to improve lip-sync accuracy while preserving semantic content.
• Evaluation metrics such as DTW, COMET, LSE-D, and LSE-C help balance synchronization precision with the overall intelligibility and realism of generated speech.

Searching arXiv for recent and foundational papers on phonetic synchronization and closely related synchronization formulations in speech, dubbing, VSR, and talking-face generation. Phonetic Synchronization (PS) denotes a family of alignment problems in which phonetic or articulatory structure is made temporally consistent across representations such as source and target text in automated dubbing, video frames and acoustic units in visual speech recognition, or audio and lip motion in talking-face generation. In recent arXiv literature, the term does not name a single universal formalism; rather, it refers to several closely related mechanisms that all attempt to preserve or recover the timing of speech-relevant events beyond coarse sentence- or utterance-level alignment. The common theme is that duration matching alone is insufficient: systems must also preserve the phonetic pattern that governs visible mouth shape, local acoustic realization, or crossmodal correspondence (Hong et al., 10 Apr 2026, Ahn et al., 2024, Park et al., 2023).

1. Scope and principal senses of the term

Within speech and audiovisual generation, PS is used in at least three technically distinct but related senses. In automated dubbing, it is a text-selection strategy that chooses paraphrases whose vowel pronunciations are close to those of the source, so that synthesized target speech better matches the original lip motion without video manipulation. In visual speech recognition (VSR), it is a frame-level crossmodal supervision signal that forces visual representations to predict discrete audio tokens. In talking-face generation, it is the use of phonetic context and synchronization losses to generate lip motion that reflects co-articulation rather than isolated-phone articulation (Hong et al., 10 Apr 2026, Ahn et al., 2024, Park et al., 2023).

Research setting	Synchronized entities	Representative mechanism
Automated dubbing	Source and target vowel patterns under timing constraints	Isochrony followed by DTW with vowel-distance local costs
Visual speech recognition	Video frames and quantized audio tokens	Frame-level crossmodal audio token synchronization
Talking-face generation	Contextualized phones and lip motion	Audio-to-Lip masked learning plus sync losses

A recurring misconception is to treat PS as synonymous with generic temporal synchronization. The literature distinguishes several levels. Prosodic phrase synchronization in machine dubbing is explicitly described as an upstream step rather than full phoneme-by-phoneme lip synchronization, and the authors identify phoneme-level alignment, including “alignment of open, closed phones,” as future work. Conversely, audiovisual lip-to-speech work treats synchronization as estimation and correction of dataset and model offsets rather than candidate selection in text space (Öktem et al., 2019, Niu et al., 2023).

2. Automated dubbing: isochrony, vowel matching, and PS-Comet

The most explicit recent formulation appears in “PS-TTS: Phonetic Synchronization in Text-to-Speech for Achieving Natural Automated Dubbing” (Hong et al., 10 Apr 2026). That work separates dubbing synchronization into two stages. The first is isochrony, which paraphrases translated text so that target speech duration matches source speech duration. The second is PS, which preserves lip-sync by selecting target text whose vowel pronunciations are similar to the source. The motivation is that conventional automated dubbing may satisfy translation and timing constraints yet still appear visually unsynchronized on the lips, especially for linguistically distant language pairs such as Korean and English.

The pipeline begins with translation by NMT, followed by phonemization and tokenization. Source and target phoneme sequences are denoted

$$S = \{S_1, \ldots, S_N\}, \qquad T = \{t_1, \ldots, t_M\},$$

and after blank-token insertion,

$$S_p = \{b, S_1, b, S_2, \ldots, b, S_N, b\}, \qquad T_b = \{b, t_1, b, t_2, \ldots, b, t_M, b\}.$$

Pause detection is described through RMS-based energy thresholding and a CTC-based forced alignment alternative,

$L_{\mathrm{CTC}} = -\log [p(y|x)]. \tag{1}$

After pause removal, refined frame count and speaking rate are defined as

$K' = K - \sum_{i=0}^{P} r_i, \tag{2}$

$$R(S_b) = \frac{K'}{\text{duration}_{\mathrm{TTS}(S_b)}}. \tag{3}$$

Isochrony predicts target duration and enforces semantic similarity using LaBSE:

$$K(T_b) = R(S_b)\times \text{duration}_{\mathrm{TTS}(T_b)}, \tag{4}$$

$$C_{\mathrm{sm}}(S_b, T_b) = \cos\big(\Phi(S_b), \Phi(T_b)\big). \tag{5}$$

A candidate is accepted if its duration is within 30 frames of the source and its cosine similarity is at least 0.75; the paper also states this equivalently as

$$K' - 26 \le R(T_b) \le K' + 26,\qquad C_{\mathrm{sm}}(S_b, T_b)\ge 0.75. \tag{6–7}$$

If those conditions fail, GPT-4 generates paraphrases iteratively, up to 60 candidates.

PS proper is vowel-centric. Using a cross-lingual TTS training corpus containing 6,000 Korean and 5,000 English sentences, the paper extracts vowel representations from the transformer-based text encoder, clusters each vowel into five centroids via K-means, and averages them into a vowel prototype,

$H_i = \frac{1}{N}\sum_{j=1}^{N} v_{ij}, \tag{8}$

with $N=5$. Cross-lingual vowel distance is then

$$S_p = \{b, S_1, b, S_2, \ldots, b, S_N, b\}, \qquad T_b = \{b, t_1, b, t_2, \ldots, b, t_M, b\}.$$0

After duration expansion of source and candidate phoneme sequences, non-vowels are mapped to a null index and dynamic time warping with a Sakoe-Chiba constraint is applied:

$$S_p = \{b, S_1, b, S_2, \ldots, b, S_N, b\}, \qquad T_b = \{b, t_1, b, t_2, \ldots, b, t_M, b\}.$$1

$$S_p = \{b, S_1, b, S_2, \ldots, b, S_N, b\}, \qquad T_b = \{b, t_1, b, t_2, \ldots, b, t_M, b\}.$$2

This mechanism selects, among already isochronous paraphrases, the candidate whose vowel timing pattern is closest to the source’s articulatory pattern.

The main limitation of PS in this form is semantic drift. The paper therefore introduces PS-Comet, implemented as PS-Comet-TTS, and reports that DTW and COMET are only weakly and negatively correlated, with Pearson’s $$S_p = \{b, S_1, b, S_2, \ldots, b, S_N, b\}, \qquad T_b = \{b, t_1, b, t_2, \ldots, b, t_M, b\}.$$3 and Spearman’s $$S_p = \{b, S_1, b, S_2, \ldots, b, S_N, b\}, \qquad T_b = \{b, t_1, b, t_2, \ldots, b, t_M, b\}.$$4. PS-Comet combines normalized and inverted DTW with COMET:

$$S_p = \{b, S_1, b, S_2, \ldots, b, S_N, b\}, \qquad T_b = \{b, t_1, b, t_2, \ldots, b, t_M, b\}.$$5

The chosen weights are $$S_p = \{b, S_1, b, S_2, \ldots, b, S_N, b\}, \qquad T_b = \{b, t_1, b, t_2, \ldots, b, t_M, b\}.$$6 and $$S_p = \{b, S_1, b, S_2, \ldots, b, S_N, b\}, \qquad T_b = \{b, t_1, b, t_2, \ldots, b, t_M, b\}.$$7. Across Korean, English, and French language pairs, PS-Comet is reported as the best-performing variant for balancing lip-sync accuracy with semantic preservation (Hong et al., 10 Apr 2026).

3. Precursors: speech–text alignment and prosodic phrase synchronization

Earlier work addressed related alignment problems at coarser granularity. “Alignment of Speech to Highly Imperfect Text Transcriptions” aligns speech to highly imperfect ASR transcripts by detecting a subset of phonemes in the speech track and aligning them to the sequence of phonemes extracted from the transcript. On four speech-transcript sets ranging from 22 to 108 minutes, the reported correct matching of phonemes within 10, 20, and 30 second error margins is more than 60%, 75%, and 90% of text, respectively, on average. This is a temporal speech–text alignment problem rather than lip-sync, but it establishes phoneme sequence matching as a practical strategy under noisy transcription conditions [0612139].

Machine dubbing work then moved from subtitle-timestamp synchronization toward phrase-structured alignment. “Prosodic Phrase Alignment for Machine Dubbing” defines prosodic phrases as “voiced segments terminated with a silent pause” and focuses on cross-lingual phrasing rather than full phoneme-level synchronization. Source phrase boundaries are inferred from pauses, transferred to target tokens via NMT attention, and then used to condition TTS durations. Candidate target phrase labeling is scored by masked attention,

$$S_p = \{b, S_1, b, S_2, \ldots, b, S_N, b\}, \qquad T_b = \{b, t_1, b, t_2, \ldots, b, t_M, b\}.$$8

with

$$S_p = \{b, S_1, b, S_2, \ldots, b, S_N, b\}, \qquad T_b = \{b, t_1, b, t_2, \ldots, b, t_M, b\}.$$9

and selected by

$L_{\mathrm{CTC}} = -\log [p(y|x)]. \tag{1}$0

A pause threshold of $L_{\mathrm{CTC}} = -\log [p(y|x)]. \tag{1}$1 is used to distinguish articulatory pauses from linguistically motivated phrase breaks. In evaluation, the aligned phrases achieve an average speech rate ratio of 1.27, close to the original corpus ratio of 1.31. Human ratings show better lip-sync for the synchronized system than for subtitle-style dubbing, although translation quality remains lower (Öktem et al., 2019).

Taken together, these studies suggest a progression from coarse phoneme sequence alignment, to phrase-level timing and pausing, to the vowel- and articulation-level selection criteria used in recent PS-TTS systems. That implication is interpretive rather than explicitly stated in any single paper.

4. Crossmodal PS in visual speech recognition and lip-to-speech synthesis

In VSR, PS is formulated as dense crossmodal supervision. “SyncVSR: Data-Efficient Visual Speech Recognition with End-to-End Crossmodal Audio Token Synchronization” treats PS as the central mechanism for reducing homophene ambiguity by synchronizing each video frame with quantized audio tokens rather than only with text labels or masked latent targets. The paper uses a fixed alignment of 25 fps video and 16 kHz audio, stated as 1 video frame : 4 quantized audio tokens at a 100 Hz audio-token rate. The synchronization loss is

$L_{\mathrm{CTC}} = -\log [p(y|x)]. \tag{1}$2

and the total loss is

$L_{\mathrm{CTC}} = -\log [p(y|x)]. \tag{1}$3

The paper characterizes this as “crossmodal audio token synchronization,” with full-length non-autoregressive token prediction from all frames. On LRS2 ablation, adding Sync yields a much larger improvement than adding CTC alone; on LRW and LRW-WB, the reported top-1 accuracies are 93.2% and 95.0%, and the method is said to reduce data usage by up to ninefold (Ahn et al., 2024).

Lip-to-speech synthesis addresses a related but distinct synchronization problem. “On the Audio-visual Synchronization for Lip-to-Speech Synthesis” separates data asynchrony, where paired video $L_{\mathrm{CTC}} = -\log [p(y|x)]. \tag{1}$4 and audio $L_{\mathrm{CTC}} = -\log [p(y|x)]. \tag{1}$5 are offset by $L_{\mathrm{CTC}} = -\log [p(y|x)]. \tag{1}$6, from model asynchrony, where generated speech is shifted by $L_{\mathrm{CTC}} = -\log [p(y|x)]. \tag{1}$7. The proposed synchronized lip-to-speech model, SLTS, incorporates an Automatic Synchronization Mechanism composed of a Data Synchronization Module and a Self-Synchronization Module. Both use an audio-visual time offset predictor with mel-spectrogram-frame resolution, also at 100 Hz. The offset distribution is

$L_{\mathrm{CTC}} = -\log [p(y|x)]. \tag{1}$8

and the self-synchronization loss is

$L_{\mathrm{CTC}} = -\log [p(y|x)]. \tag{1}$9

The paper argues that standard metrics such as STOI, ESTOI, and MCD are highly sensitive to time alignment and therefore misleading on asynchronous test data; it introduces an audio alignment frontend that searches offsets from $K' = K - \sum_{i=0}^{P} r_i, \tag{2}$0 ms to $K' = K - \sum_{i=0}^{P} r_i, \tag{2}$1 ms in 10 ms steps before computing alignment-sensitive metrics (Niu et al., 2023).

These formulations share a dense temporal view of PS. In SyncVSR, the synchronized objects are video frames and discrete acoustic tokens. In SLTS, they are visual embeddings and mel-spectrogram frames. A plausible implication is that phonetic synchronization in crossmodal speech systems is moving away from coarse semantic alignment toward explicit frame-level or token-level correspondence.

5. Co-articulation and phonetic context in talking-face generation

Talking-face generation introduces another interpretation of PS: synchronization must be conditioned on phonetic context because isolated phones do not determine lip shape. “Exploring Phonetic Context-Aware Lip-Sync For Talking Face Generation” states that articulation varies with preceding and following phones due to co-articulation and argues that common methods using only a short audio window of about 0.2 seconds are insufficient for lip synchronization. The proposed Context-Aware Lip-Sync framework (CALS) therefore learns contextualized lip motion units rather than fixed phone-to-lip mappings (Park et al., 2023).

CALS comprises an Audio-to-Lip module and a Lip-to-Face module. Audio is represented as a sequence of frame-level mel-spectrogram units,

$K' = K - \sum_{i=0}^{P} r_i, \tag{2}$2

with audio unit size 80 × 4, sampled at 16 kHz, window size 400, and hop size 160. Masked learning corrupts the input as

$K' = K - \sum_{i=0}^{P} r_i, \tag{2}$3

where continuous sequences of $K' = K - \sum_{i=0}^{P} r_i, \tag{2}$4 audio units are masked and 50% of audio units are randomly masked. The Audio-to-Lip module predicts contextualized lip motion units,

$K' = K - \sum_{i=0}^{P} r_i, \tag{2}$5

and is trained against lip motion units extracted by a pretrained visual sync encoder,

$K' = K - \sum_{i=0}^{P} r_i, \tag{2}$6

The masked objective is

$K' = K - \sum_{i=0}^{P} r_i, \tag{2}$7

After face generation, synchronization is further enforced through

$K' = K - \sum_{i=0}^{P} r_i, \tag{2}$8

with

$K' = K - \sum_{i=0}^{P} r_i, \tag{2}$9

and

$$R(S_b) = \frac{K'}{\text{duration}_{\mathrm{TTS}(S_b)}}. \tag{3}$$0

The total loss is

$$R(S_b) = \frac{K'}{\text{duration}_{\mathrm{TTS}(S_b)}}. \tag{3}$$1

with $$R(S_b) = \frac{K'}{\text{duration}_{\mathrm{TTS}(S_b)}}. \tag{3}$$2, $$R(S_b) = \frac{K'}{\text{duration}_{\mathrm{TTS}(S_b)}}. \tag{3}$$3, and $$R(S_b) = \frac{K'}{\text{duration}_{\mathrm{TTS}(S_b)}}. \tag{3}$$4.

Empirically, the paper reports that the effective window size for lip generation is approximately 1.2 seconds. On LRS2, the proposed method reports PSNR 32.603, SSIM 0.876, LMD 1.056, LSE-D 5.337, and LSE-C 9.225; compared with Wav2Lip, LMD improves from 1.519 to 1.056, LSE-D from 5.895 to 5.337, and LSE-C from 8.795 to 9.225. The ablation identifies the Audio-to-Lip module as the largest contributor to synchronization improvement. The conceptual formulation is succinct: a phone should be mapped not to a fixed lip pose, but to a context-dependent lip-motion unit shaped by neighboring phones (Park et al., 2023).

6. Evaluation criteria, trade-offs, and terminological boundaries

Evaluation of PS depends strongly on which object is being synchronized. In automated dubbing, the principal objective metrics reported are LSE-D, LSE-C, and UTMOS, with semantic similarity additionally assessed using LaBSE, SBERT, and LASER. On Korean and English lip-reading datasets, baseline TTS reports LSE-D 12.671 and LSE-C 1.128, ISO+PS reports 12.378 and 1.175, and ISO+PS-Comet reports 12.175 and 1.404; UTMOS improves from 2.453 for baseline TTS to 2.562 for PS and 2.614 for PS-Comet. The same work reports 98.229 VMAF for PS-Comet-TTS versus 86.208 for deepfake, and runtime of 1 min 34 s for a 10 s video versus 19 min 29 s for deepfake processing. These figures frame the central trade-off as lip-sync versus semantic preservation, with PS-Comet introduced specifically because PS alone may select acoustically favorable but semantically weaker paraphrases (Hong et al., 10 Apr 2026).

In crossmodal generation and recognition, alignment-sensitive metrics themselves become part of the methodological debate. Lip-to-speech work argues that STOI, ESTOI, and MCD can degrade sharply under even small offsets and therefore proposes a time-alignment frontend before computing them. Talking-face work uses LMD, LSE-D, and LSE-C; lower LMD and LSE-D and higher LSE-C indicate better synchronization. This suggests that PS research is inseparable from evaluation design: a system that is more aligned to the video may be penalized if the reference itself is asynchronous (Niu et al., 2023, Park et al., 2023).

The term also requires terminological discipline. In tonal ASR, a related but different synchronization question concerns whether phones and tones should be modeled synchronously on a joint tier or asynchronously on separate tiers. There, synchronous modeling gives lower joint phone+tone error rate, while asynchronous training gives lower tone error rate. Outside speech technology, “PS” frequently denotes phase synchronization, as in music-signal analysis and neural-network dynamics; those usages concern dynamical phase locking rather than phonetic or articulatory alignment (Li et al., 2020, Mukherjee et al., 2014, Yamakou et al., 2023).

Across these literatures, a consistent conclusion emerges: synchronization at the level of utterance duration or subtitle timestamps is not sufficient for natural audiovisual speech. Whether implemented by vowel-aware paraphrase selection, frame-level audio token prediction, context-aware lip-motion units, or phoneme sequence alignment, PS seeks to preserve the temporal structure of speech events that govern intelligibility, visible articulation, and crossmodal coherence.