Cascade-Free Mandarin Visual Speech Recognition via Semantic-Guided Cross-Representation Alignment

Abstract: Chinese mandarin visual speech recognition (VSR) is a task that has advanced in recent years, yet still lags behind the performance on non-tonal languages such as English. One primary challenge arises from the tonal nature of Mandarin, which limits the effectiveness of conventional sequence-to-sequence modeling approaches. To alleviate this issue, existing Chinese VSR systems commonly incorporate intermediate representations, most notably pinyin, within cascade architectures to enhance recognition accuracy. While beneficial, in these cascaded designs, the subsequent stage during inference depends on the output of the preceding stage, leading to error accumulation and increased inference latency. To address these limitations, we propose a cascade-free architecture based on multitask learning that jointly integrates multiple intermediate representations, including phoneme and viseme, to better exploit contextual information. The proposed semantic-guided local contrastive loss temporally aligns the features, enabling on-demand activation during inference, thereby providing a trade-off between inference efficiency and performance while mitigating error accumulation caused by projection and re-embedding. Experiments conducted on publicly available datasets demonstrate that our method achieves superior recognition performance.

• The paper introduces a cascade-free multitask Conformer that jointly predicts characters, IPA phonemes, and visemes, using semantic-guided local contrastive alignment to avoid cascading errors and latency.
• The model achieves 20.38% CER on seen speakers and 38.23% on unseen speakers in CMLR, outperforming video-only and landmark-augmented baselines while using video input alone.
• The alignment loss is essential, raising unseen-speaker CER from 38.23% to 42.13% when removed, while on-demand branches enable a latency–accuracy trade-off from 79.1 ms visual-only inference to 95.2 ms with all representations.

Motivation and problem statement

Mandarin visual speech recognition (VSR) lags behind English VSR largely because Mandarin is a tonal, non-phonetic language: there is no direct orthographic-to-articulatory correspondence between characters and visual/acoustic signals, and the character vocabulary is large. Prior systems therefore insert intermediate representations—most commonly pinyin—into cascaded sequence-to-sequence pipelines, where a first stage predicts pinyin and a second stage maps pinyin to characters. The authors identify two structural defects in this design: error accumulation, since an incorrect pinyin hypothesis propagates into character prediction even when shortcut connections or cross-stage attention are used; and inference latency, since strict stage-wise dependency forbids parallel execution. They additionally note that projection and re-embedding operations between stages introduce further errors and potential gradient discontinuities during training.

The paper proposes a cascade-free multitask architecture that jointly learns phoneme and viseme intermediate representations alongside characters, with a semantic-guided local contrastive loss aligning the two auxiliary representations temporally. Intermediate representations are activated on demand at inference, yielding an explicit accuracy–latency trade-off without any stage-wise dependency (2603.21808).

Method

Linguistic supervision. Rather than pinyin characters or initial–finals used in prior work, the method adopts IPA-based phonemes, which more faithfully reflect speech production, and visemes—the visual counterparts grouping phonemes sharing lip configurations. A 15-way viseme inventory is defined over Mandarin IPA symbols with frequency statistics; pypinyin generates phoneme/viseme sequences from character labels during preprocessing. This mapping provides structured semantic guidance for cross-representation alignment while the contrastive objective still learns soft temporal correspondences within local windows.

Architecture. Visual features are extracted by 3D convolutions for local spatiotemporal dynamics followed by a pretrained ResNet frame-wise extractor. Dedicated three-layer phoneme and viseme encoders with MLP heads are trained with CTC loss for intrinsic temporal alignment. A multi-branch DropPath fusion layer stochastically masks and rescales each branch before aggregation:

$$\tilde{F} = \phi\left(F + (P \odot B_p) + (V \odot B_v)\right), \quad B_p, B_v \sim \text{Bernoulli}(1-p_{drop})$$

The fused features feed a Conformer-style character encoder/decoder trained with hybrid CTC/Attention loss.

Semantic-guided local contrastive loss. A binary matrix $M$ encodes phoneme-to-viseme semantic compatibility from predicted logits; a window mask $W$ restricts contrast to temporally adjacent tokens ($|i-j| \le r$, $r = \lfloor w/2 \rfloor$, $w=5$). Cosine similarities between L2-normalized viseme and phoneme tokens are temperature-scaled into a local distribution $q(j|i)$, matched against the positive-mask distribution $p(j|i)$ via KL divergence, averaged over tokens possessing at least one positive pair. The total loss combines the hybrid CTC/Attention character loss, the alignment loss, and CTC losses on both auxiliary branches with weighting hyperparameters $\lambda_1, \lambda_2$.

On-demand inference. Because all branches are learned jointly, inference can use visual features alone ($F$), or augment them with phoneme ($M$0), viseme ($M$1), or both ($M$2). No inter-stage dependency exists, so branches process in parallel.

Experimental results

Experiments use the public CMLR dataset (>100k sentences, 11 speakers), evaluated under seen-speaker (official 7:1:2 split) and unseen-speaker (s2/s6 held out) protocols with Character Error Rate (CER).

Method	Modality	Seen CER (%)	Unseen CER (%)
CTCH-LipNet	V	22.02	62.47
CT-MIR-Net	V	21.45	51.38
LipFormer	V+L	27.79	43.18
GusLip	V+L	29.98	43.61
Proposed	V	20.38	38.23

Two claims stand out. First, the video-only model outperforms all prior video-only baselines on both splits, with particularly large gains unseen (38.23% vs. 43.18% for the best landmark-augmented competitor). Second—and more assertively—it surpasses systems that consume additional facial landmark modalities despite using video alone, which the authors take as evidence that the architecture captures discriminative cues without supplementary inputs.

Ablations. Removing $M$3 degrades CER from 20.38% to 24.34% (seen) and 38.23% to 42.13% (unseen), confirming the alignment loss is the principal driver of the gains. Removing phoneme/viseme supervision entirely yields 35.25%/49.78%, showing that intermediate representation learning—not merely the Conformer backbone—accounts for most of the improvement over single-stage video-to-character modeling.

Representation and efficiency analysis. With only visual features active, CER is 24.47%/45.72%; adding both branches reaches 20.38%/38.23%. Viseme alone gives marginal seen-split benefit but stronger unseen robustness, which the authors attribute to speaker-independent articulation features. On latency, the cascade-free Conformer at 95.2 ms (all branches, 129.58M parameters) is faster than the dual-stage CT-MIR-Net (160.9 ms, 122.57M params) and the three-stage CSSMCM (137.2 ms), while the visual-only configuration runs at 79.1 ms. Qualitative examples show progressive homophone correction as more branches activate—for one utterance, CER drops from 0.4545 ($M$4) to 0.0909 ($M$5)—though residual homophone errors persist even with full activation.

Limitations and open questions

The paper concedes several points directly. Homophone disambiguation remains incomplete: phonetically similar syllables with different tones still produce errors even with all representations active, indicating that local acoustic-visual correspondence is insufficient and richer contextual reasoning is needed—an open question the paper poses but does not resolve. The seen/unseen gap remains substantial (roughly 18 CER points absolute), and the authors suggest identity disentanglement as future work rather than addressing it here. The evaluation is confined to CMLR, a constrained 11-speaker dataset; generalization to larger or in-the-wild corpora is not demonstrated. Finally, the phoneme/viseme labels are generated automatically via pypinyin from character transcripts, so label noise from this pipeline is not quantified.

Conclusion

This work replaces cascaded pinyin-mediated Mandarin VSR with a cascade-free multitask Conformer that jointly supervises phoneme and viseme branches, aligned through a semantic-guided local contrastive loss over a linguistically grounded phoneme-to-viseme map. It achieves state-of-the-art CER on CMLR (20.38% seen, 38.23% unseen) using video only, outperforming even landmark-augmented baselines, while offering controllable accuracy–latency trade-offs at inference. The remaining homophone ambiguity and the sizable unseen-speaker gap define the concrete open problems left by the paper.