HM-Talker: Speech Recog & Talking-Head Synthesis

• HM-Talker is a term used for two distinct systems: a speaker-mask transformer for multi-talker ASR and a hybrid-motion framework for audio-driven talking head synthesis.
• In the ASR variant, the system integrates speaker-aware autoregressive transcription with a dedicated mask branch and quantized timestamp tokens to robustly segment overlapped speech.
• For talking head synthesis, HM-Talker fuses implicit audio cues with explicit Action Units via cross-modal disentanglement and gated fusion to reduce lip jitter and phoneme-viseme misalignment.

Searching arXiv for HM-Talker and closely related papers to ground the article. arxiv_search.query({"^^^^^^^^5search_query5 OR ti:\5^^^^^^^^"HM-Talker\" OR abs:\5^^^^^^^^"speaker mask transformer\"^^^^^^^^5 OR ti:\5^^^^^^^^"LLM Can Transcribe Speech in Multi-Talker Scenarios with Versatile Instructions\"","start":^^^^5search_query5^^^^,"max_results":^^^^5all:\5search_query5 I found the relevant arXiv entries for HM-Talker and adjacent multi-talker research, including the talking-head synthesis paper "HM-Talker: Hybrid Motion Modeling for High-Fidelity Talking Head Synthesis" (&&&^^^^5search_query5^^^^&&&), the multi-talker speech-recognition paper summarized as HM-Talker / speaker-mask transformer (&&&^^^^5all:\5^^^^&&&), and related multi-talker ASR work such as MT-LLM (&&&^^^^5 OR ti:\5^^^^&&&), Whisper-based multilingual multi-talker transfer (&&&^^^^5 OR abs:\5^^^^&&&), and joint multi-/target-talker Whisper adaptation (&&&^^^^5 OR ti:\5^^^^&&&). HM-Talker is a name that has been used in arXiv literature for two distinct research systems in different subfields. In speech processing, it denotes a transformer-based method for multi-talker overlapped speech recognition that combines speaker-aware autoregressive transcription with a dedicated speaker mask branch for diarization-oriented segment detection (&&&^^^^5all:\5^^^^&&&). In talking-face generation, it denotes "HM-Talker: Hybrid Motion Modeling for High-Fidelity Talking Head Synthesis," an audio-driven talking head framework that combines implicit audio motion cues with explicit Action Unit (AU) priors through cross-modal disentanglement and hybrid fusion (&&&^^^^5search_query5^^^^&&&). The shared label therefore does not identify a single canonical architecture; rather, it refers to separate systems concerned with speech-related human communication signals in either acoustic or visual form.

^^^^5all:\5^^^^. Nomenclature and scope

The speech-recognition usage of HM-Talker arises from the paper "Speaker Mask Transformer for Multi-talker Overlapped Speech Recognition" (&&&^^^^5all:\5^^^^&&&). In that work, HM-Talker refers to the proposed speaker-mask transformer method for multi-talker overlapped speech recognition. Its stated objective is not only to recognize the lexical content of overlapped speech, but also to address speaker diarization by answering WHAT each speaker said, WHO said it, and WHEN each speaker was active (&&&^^^^5all:\5^^^^&&&).

A separate usage appears in the later paper "HM-Talker: Hybrid Motion Modeling for High-Fidelity Talking Head Synthesis" (&&&^^^^5search_query5^^^^&&&). There, HM-Talker is an audio-driven talking head synthesis framework that targets motion blur, lip jitter, and phoneme-viseme misalignment by introducing a hybrid motion representation composed of implicit and explicit motion cues (&&&^^^^5search_query5^^^^&&&).

A common misconception is to treat HM-Talker as a single model family. The arXiv record instead supports a narrower and more precise view: the name has been independently attached to two technically unrelated systems, one for overlapped speech recognition and diarization, and one for talking head video generation (&&&^^^^5all:\5^^^^&&&, &&&^^^^5search_query5^^^^&&&).

^^^^5 OR ti:\5^^^^. HM-Talker in multi-talker overlapped speech recognition

In the speech-recognition setting, HM-Talker is formulated as a joint ASR and speaker diarization system built on Whisper base, with 7^^^^5 OR ti:\5^^^^M parameters, 6 transformer blocks in encoder and decoder, hidden size: 5^^^^5all:\5 OR ti:\5^^^^, and 8^^^^5search_query5^^^^-channel log-magnitude Mel spectrograms using ^^^^5 OR ti:\55^^^^ ms windows and ^^^^5all:\5search_query5^^^^ ms stride (&&&^^^^5all:\5^^^^&&&). The method modifies autoregressive transformer ASR by adding speaker labels to the serialized output and by attaching a speaker mask branch to the encoder representation.

The speaker-labeled output is introduced through token sequences such as

PRESERVED_PLACEHOLDER_^^^^5search_query5^^^^

and

PRESERVED_PLACEHOLDER_^^^^5all:\5^^^^

The paper quantizes timestamps with ^^^^5 OR ti:\5search_query5^^^^ ms resolution, following Whisper-style timestamp tokens (&&&^^^^5all:\5^^^^&&&). Its ASR loss is written as

PRESERVED_PLACEHOLDER_^^^^5 OR ti:\5^^^^

with PRESERVED_PLACEHOLDER_^^^^5 OR abs:\5^^^^ the acoustic feature sequence and PRESERVED_PLACEHOLDER_^^^^5 OR ti:\5^^^^ the model parameters (&&&^^^^5all:\5^^^^&&&).

The central HM-Talker innovation is the speaker mask branch, inspired by Mask R-CNN. The ASR branch still predicts serialized text with speaker labels, but diarization is shifted to a separate branch that predicts the speech activity mask of each speaker from encoder hidden states. For speaker $k$, the mask loss is

$L^k_{\text{mask} = -\frac{1}{D}\sum_{i=1}^{D} \left[ y_i \log(p_i) + (1 - y_i) \log(1 - p_i) \right]$

where $y_i \in \{0,1\}$ is the ground-truth mask label and $p_i$ is the sigmoid output (&&&^^^^5all:\5^^^^&&&). These mask targets are constructed using an energy-based VAD over ^^^^5 OR ti:\5search_query5^^^^ ms segments, with time-alignment timestamps from the Montreal Forced Aligner used to anchor speech starts (&&&^^^^5all:\5^^^^&&&).

The joint training objective is

$L_{\text{SPK-MASK} = (1-\lambda) * L_{ASR} + \lambda \sum_{k=1}^{K} L^k_{mask}$

and, for the proposed model, the decoder target removes timestamps and keeps speaker labels:

PRESERVED_PLACEHOLDER_^^^^5all:\5search_query5^^^^

The paper states that the mask branch is optimized only when the main branch outputs speaker labels, tying diarization learning directly to speaker-aware transcription (&&&^^^^5all:\5^^^^&&&).

This design is motivated by a specific failure mode of timestamp-token diarization. If one speaker is interrupted and later resumes, start/end timestamps alone can represent the serialized utterances awkwardly. The speaker mask branch instead learns a speaker-specific activity map from encoder features, which can indicate all speech frames belonging to a speaker even when those frames are non-contiguous in the output sequence (&&&^^^^5all:\5^^^^&&&). This suggests that HM-Talker treats transcription and segmentation as related but not identical prediction problems.

^^^^5 OR abs:\5^^^^. Architectural variants, data, and empirical behavior of the speech-recognition system

The paper evaluates several mask-head configurations: L-FC, L-FC-CNN, CA-FC, and CA-FC-CNN. The CNN stack uses ^^^^5 OR abs:\5 OR ti:\5^^^^ and 6^^^^5 OR ti:\5^^^^ kernels, kernel size PRESERVED_PLACEHOLDER_^^^^5all:\5all:\5^^^^, stride ^^^^5all:\5^^^^, and dropout ^^^^5search_query5^^^^.^^^^5 OR ti:\55^^^^ after the second CNN (&&&^^^^5all:\5^^^^&&&). The reported finding is that the cross-attention + CNN designs generally work best for diarization (&&&^^^^5all:\5^^^^&&&).

Training uses all 96^^^^5search_query5^^^^ hours of LibriSpeech training data and constructs two overlap regimes. Case ^^^^5all:\5^^^^ is simple two-speaker overlap created by randomly selecting an utterance from a different speaker and mixing at ^^^^5search_query5^^^^ dB SIR. Case ^^^^5 OR ti:\5^^^^ is a more complex overlap pattern with utterance order Speaker ^^^^5all:\5^^^^, Speaker ^^^^5 OR ti:\5^^^^, Speaker ^^^^5all:\5^^^^, and overlap duration randomly sampled from ^^^^5search_query5^^^^ to 5 seconds. The resulting training sets are train96^^^^5search_query5^^^^-Org, train96^^^^5search_query5^^^^-Set^^^^5all:\5, and train96^^^^5search_query5^^^^-Set^^^^5 OR ti:\5^^^^ with ratios ^^^^5all:\5^^^^:^^^^5all:\5 and ^^^^5all:\5^^^^:^^^^5all:\5 as specified in the data (&&&^^^^5all:\5^^^^&&&). Evaluation is performed on Set^^^^5all:\5^^^^-^^^^5all:\5, Set^^^^5all:\5^^^^-^^^^5 OR abs:\5^^^^s, and Set^^^^5 OR ti:\5^^^^-^^^^5all:\5^^^^s derived from test-clean, with diarization measured by DER with ^^^^5search_query5^^^^.^^^^5 OR ti:\5^^^^ collar, speaker counting by SCA, and recognition by WER (&&&^^^^5all:\5^^^^&&&).

Representative results show the trade-off between timestamp-token methods and the mask branch. Under Case ^^^^5all:\5^^^^, SPK gives Set^^^^5all:\5^^^^-^^^^5all:\5 WER: 5.9^^^^5 OR abs:\5^^^^, SCA: 99.5^^^^5search_query5^^^^ and Set^^^^5all:\5^^^^-^^^^5 OR abs:\5^^^^s WER: 9.^^^^5 OR ti:\5^^^^7, SCA: 99.^^^^5search_query5^^^^9. Timestamp-based SPK-TS-^^^^5all:\5^^^^ reduces diarization error but worsens WER, with Set^^^^5all:\5^^^^-^^^^5all:\5 WER: 8.^^^^5all:\5 OR ti:\5^^^^, DER: ^^^^5 OR ti:\5^^^^.7^^^^5 OR ti:\5^^^^, SCA: 99.89 and Set^^^^5all:\5^^^^-^^^^5 OR abs:\5^^^^s WER: ^^^^5all:\5 OR abs:\5^^^^.^^^^5all:\5 OR abs:\5^^^^, DER: ^^^^5 OR abs:\5^^^^.^^^^5search_query5^^^^9, SCA: 99.9^^^^5search_query5^^^^. By contrast, Mask: CA-FC-CNN (^^^^5search_query5^^^^.5) achieves Set^^^^5all:\5^^^^-^^^^5all:\5 WER: 6.^^^^5search_query5 OR abs:\5^^^^, DER: ^^^^5all:\5^^^^.5^^^^5 OR ti:\5^^^^, SCA: 99.69 and Set^^^^5all:\5^^^^-^^^^5 OR abs:\5^^^^s WER: 9.9^^^^5all:\5^^^^, DER: ^^^^5 OR ti:\5^^^^.^^^^5search_query5 OR ti:\5^^^^, SCA: 99.65 (&&&^^^^5all:\5^^^^&&&).

The strongest contrast appears in Case ^^^^5 OR ti:\5^^^^, where the paper notes that SPK-TS-^^^^5 OR ti:\5^^^^ performs badly in the hardest condition, with Set^^^^5 OR ti:\5^^^^-^^^^5all:\5^^^^s DER: ^^^^5 OR ti:\5^^^^6.^^^^5 OR abs:\5^^^^7%. The proposed mask method remains low in DER: Mask: L-FC (^^^^5search_query5^^^^.5) gives Set^^^^5 OR ti:\5^^^^-^^^^5all:\5^^^^s WER: 6.59, DER: ^^^^5all:\5^^^^.^^^^5all:\5search_query5 SCA: 99.95, while CA-FC-CNN (^^^^5search_query5^^^^.5) gives Set^^^^5 OR ti:\5^^^^-^^^^5all:\5^^^^s WER: 6.5^^^^5all:\5^^^^, DER: ^^^^5search_query5^^^^.7^^^^5search_query5 SCA: 99.85 (&&&^^^^5all:\5^^^^&&&). With Whisper Large-v^^^^5 OR ti:\5^^^^, the paper reports that ASR WER improves significantly while DER stays low; for example, LargeV^^^^5 OR ti:\5^^^^, Mask: L-FC (^^^^5search_query5^^^^.5) gives Set^^^^5all:\5^^^^-^^^^5all:\5 DER: ^^^^5search_query5^^^^.76 and Set^^^^5 OR ti:\5^^^^-^^^^5all:\5^^^^s DER: ^^^^5search_query5^^^^.8^^^^5search_query5 (&&&^^^^5all:\5^^^^&&&).

The paper also studies the loss balance PRESERVED_PLACEHOLDER_^^^^5all:\5 OR ti:\5^^^^. It reports that small PRESERVED_PLACEHOLDER_^^^^5all:\5 OR abs:\5^^^^ gives better ASR and weaker diarization, large PRESERVED_PLACEHOLDER_^^^^5all:\5 OR ti:\5^^^^ gives stronger diarization and more ASR degradation, and PRESERVED_PLACEHOLDER_^^^^5all:\55^^^^ gives the best trade-off (&&&^^^^5all:\5^^^^&&&). This supports the interpretation that diarization supervision is helpful but can interfere with lexical modeling if weighted too heavily.

Within the broader multi-talker ASR literature, HM-Talker occupies a distinct position. Whisper adaptation with enhanced Serialized Output Training (SOT) and timestamps has been used to jointly model multi-talker ASR, speaker counting, and utterance timestamp prediction in meeting transcription (&&&^^^^5 OR abs:\5^^^^&&&). Whisper has also been extended with a Sidecar Separator, Target Talker Identifier, and soft prompt tuning to jointly address multi-talker and target-talker ASR (&&&^^^^5 OR ti:\5^^^^&&&). MT-LLM reformulates overlapped speech transcription as instruction-conditioned language generation, supporting multi-talker ASR, target-talker ASR, sex-specific ASR, order-specific ASR, target-lingual ASR, and keyword-tracing ASR (&&&^^^^5 OR ti:\5^^^^&&&). Relative to those directions, HM-Talker is specifically focused on the claim that timestamp tokens alone are brittle for diarization in complex overlap and that explicit mask prediction offers a better solution (&&&^^^^5all:\5^^^^&&&).

^^^^5 OR ti:\5^^^^. HM-Talker in audio-driven talking head synthesis

The later HM-Talker is an audio-driven talking head generation framework targeting motion blur, lip jitter, and phoneme-viseme misalignment (&&&^^^^5search_query5^^^^&&&). Its central proposal is a hybrid motion representation that combines implicit motion cues with explicit motion cues based on Action Units (AUs). The paper states that explicit cues use anatomically defined facial muscle movements alongside implicit features to minimize phoneme-viseme misalignment (&&&^^^^5search_query5^^^^&&&).

The architecture has two named components. The Cross-Modal Disentanglement Module (CMDM) disentangles implicit and explicit motion information across audio and video modalities. CMDM takes portrait video frames PRESERVED_PLACEHOLDER_^^^^5all:\5^^^^6, audio features from an audio-visual encoder inherited from SyncTalk, OpenFace-extracted AUs, and an MLP-based audio-to-AU projection branch (&&&^^^^5search_query5^^^^&&&). It outputs four motion features:

• PRESERVED_PLACEHOLDER_^^^^5all:\5^^^^7: upper-face explicit feature.
• PRESERVED_PLACEHOLDER_^^^^5all:\5^^^^8: lower-face explicit feature from video.
• PRESERVED_PLACEHOLDER_^^^^5all:\5^^^^9: lower-face implicit audio feature.
• PRESERVED_PLACEHOLDER_^^^^5 OR ti:\5search_query5^^^^: audio-predicted explicit feature (&&&^^^^5search_query5^^^^&&&).

The explicit AU representation uses ^^^^5all:\5^^^^7 AUs split into upper-face and lower-face groups:

PRESERVED_PLACEHOLDER_^^^^5 OR ti:\5all:\5^^^^

and

PRESERVED_PLACEHOLDER_^^^^5 OR ti:\5 OR ti:\5^^^^

Upper-face explicit features are formed as

PRESERVED_PLACEHOLDER_^^^^5 OR ti:\5 OR abs:\5^^^^

while lower-face explicit features are

PRESERVED_PLACEHOLDER_^^^^5 OR ti:\5 OR ti:\5^^^^

The residual connection is used to preserve raw AU semantics while allowing nonlinear AU interactions (&&&^^^^5search_query5^^^^&&&).

The implicit audio feature is produced through AudioNet + AudioAttNet from a pre-trained audio-visual encoder feature PRESERVED_PLACEHOLDER_^^^^5 OR ti:\55^^^^, yielding PRESERVED_PLACEHOLDER_^^^^5 OR ti:\5^^^^6 (&&&^^^^5search_query5^^^^&&&). CMDM then maps audio-only features into AU-like explicit space through the Audio-to-AU Mapper:

PRESERVED_PLACEHOLDER_^^^^5 OR ti:\5^^^^7

with cross-modal alignment enforced by

PRESERVED_PLACEHOLDER_^^^^5 OR ti:\5^^^^8

This makes explicit articulatory priors available at inference even when lower-face motion is driven only by audio (&&&^^^^5search_query5^^^^&&&).

The second major component is the Hybrid Motion Modeling Module (HMMM), which fuses implicit and explicit motion cues through gated fusion:

PRESERVED_PLACEHOLDER_^^^^5 OR ti:\5^^^^9

with

PRESERVED_PLACEHOLDER_^^^^5 OR abs:\5search_query5^^^^

During training, HMMM randomly selects one of three fusion paths—audio path, masked path, and vanilla path—with default ratio

PRESERVED_PLACEHOLDER_^^^^5 OR abs:\5all:\5^^^^

The paper states that this random pairing strategy is designed to mitigate identity-dependent biases in explicit features and enforce identity-agnostic learning (&&&^^^^5search_query5^^^^&&&).

HM-Talker extends TalkingGaussian, a ^^^^5 OR abs:\5^^^^D Gaussian Splatting-based talking head system. Lower-face and upper-face features are modulated by region-specific attention from Gaussian positional encoding PRESERVED_PLACEHOLDER_^^^^5 OR abs:\5 OR ti:\5^^^^:

PRESERVED_PLACEHOLDER_^^^^5 OR abs:\5 OR abs:\5^^^^

followed by deformation prediction

PRESERVED_PLACEHOLDER_^^^^5 OR abs:\5 OR ti:\5^^^^

Final blending is written as

PRESERVED_PLACEHOLDER_^^^^5 OR abs:\55^^^^

Training follows three-stage optimization—static initialization, motion learning, and fine-tuning—with the face branch and inside-mouth branch trained in parallel for 5^^^^5search_query5^^^^,^^^^5search_query5search_query5search_query5 iterations and then jointly fine-tuned for ^^^^5all:\55^^^^,^^^^5search_query5search_query5search_query5 iterations, using Adam and AdamW, learning rate PRESERVED_PLACEHOLDER_^^^^5 OR abs:\5^^^^6, and loss weights

PRESERVED_PLACEHOLDER_^^^^5 OR abs:\5^^^^7

The total objective is

PRESERVED_PLACEHOLDER_^^^^5 OR abs:\5^^^^8

Although training uses both audio and image inputs, evaluation and deployment are audio-only for lower-face motion (&&&^^^^5search_query5^^^^&&&).

5. Evaluation profile of the talking-head system

The talking-head HM-Talker is evaluated on five public video sequences: Lieu, Jae-in, Obama, May, and Shaheen. The data contain ^^^^5 OR abs:\5^^^^ male and ^^^^5 OR ti:\5^^^^ female subjects, have average duration 7,6^^^^5 OR abs:\5^^^^7 frames, run at ^^^^5 OR ti:\55^^^^ fps, and are mostly 5^^^^5all:\5 OR ti:\5^^^^×5^^^^5all:\5 OR ti:\5^^^^ resolution, with some ^^^^5 OR ti:\55search_query5^^^^×^^^^5 OR ti:\55search_query5^^^^ videos. A ^^^^5all:\5search_query5^^^^:^^^^5all:\5 train-validation split is used (&&&^^^^5search_query5^^^^&&&).

The comparison set spans ^^^^5 OR ti:\5^^^^D methods (IP-LAP, TalkLip, DINet), NeRF-based methods (AD-NeRF, RAD-NeRF, ER-NeRF, SyncTalk), and ^^^^5 OR abs:\5^^^^DGS-based methods (GaussianTalker, TalkingGaussian) (&&&^^^^5search_query5^^^^&&&). The evaluation metrics are divided into rendering quality (PSNR, SSIM, LPIPS), motion quality (LMD, AUE-(L/U), Sync-C), lip-sync experiments with Sync-D, and efficiency in terms of training time and FPS (&&&^^^^5search_query5^^^^&&&).

For self-reconstruction, HM-Talker reports PSNR: ^^^^5 OR abs:\55^^^^.^^^^5all:\55^^^^, LPIPS: ^^^^5search_query5^^^^.^^^^5search_query5 OR ti:\5search_query5^^^^7, SSIM: ^^^^5search_query5^^^^.997^^^^5all:\5, LMD: ^^^^5 OR ti:\5^^^^.5^^^^5all:\5 OR ti:\5^^^^, AUE-L / AUE-U: ^^^^5search_query5^^^^.5^^^^5 OR abs:\5^^^^ / ^^^^5search_query5^^^^.^^^^5 OR ti:\5 OR ti:\5^^^^, Sync-C: 7.8^^^^5search_query5^^^^7, Training time: ^^^^5search_query5^^^^.5^^^^5all:\5, and FPS: ^^^^5all:\5all:\5search_query5^^^^ (&&&^^^^5search_query5^^^^&&&). The paper compares this with SyncTalk: ^^^^5 OR abs:\5 OR ti:\5^^^^.5^^^^5all:\5^^^^ PSNR, ^^^^5search_query5^^^^.^^^^5search_query5 OR ti:\5 OR ti:\5all:\5^^^^ LPIPS, 7.5^^^^5search_query5 OR ti:\5^^^^ Sync-C and TalkingGaussian: ^^^^5 OR abs:\5 OR ti:\5^^^^.^^^^5 OR ti:\5^^^^8 PSNR, ^^^^5search_query5^^^^.^^^^5search_query5 OR abs:\5search_query5^^^^9 LPIPS, 6.^^^^5 OR ti:\5 OR ti:\5^^^^6 Sync-C (&&&^^^^5search_query5^^^^&&&). For unseen audio speakers, the reported lip-sync results are Shaheen audio: Sync-D 7.59^^^^5search_query5^^^^, Sync-C 7.97^^^^5 OR ti:\5^^^^ and Lieu audio: Sync-D 7.^^^^5 OR ti:\5submittedDate5 OR ti:\5^^^^, Sync-C 7.99^^^^5 OR ti:\5^^^^ (&&&^^^^5search_query5^^^^&&&).

The ablations are used to justify the module design. Replacing CMDM with ExpNet from SadTalker causes performance to drop, especially in lip synchronization. Replacing AUs with ^^^^5 OR abs:\5^^^^DMM or BlendShape works reasonably, but AU performs best for articulation control. Replacing the inherited audio-visual encoder with DeepSpeech or HuBERT shows DeepSpeech performs worst and HuBERT is better but still below AVE. For HMMM, gated fusion is best overall, while purely explicit, purely implicit, and fixed alpha are weaker. The default three-path stochastic training ratio PRESERVED_PLACEHOLDER_^^^^5 OR abs:\5^^^^9 is also reported as best, with masking rate sampled uniformly from ^^^^5search_query5^^^^.^^^^5all:\5 to ^^^^5search_query5^^^^.^^^^5 OR abs:\5^^^^ and best masking rate around ^^^^5search_query5^^^^.^^^^5 OR ti:\5^^^^ (&&&^^^^5search_query5^^^^&&&).

The paper does not provide an extensive limitations section, but it explicitly frames several constraints through method description and experiment scope: dependence on training-time visual supervision, subject-specific portrait videos, reliance on OpenFace AU extraction, emphasis on lower-face articulation, and a secondary role for the inside-mouth branch (&&&^^^^5search_query5^^^^&&&). A plausible implication is that the method is strongest as a subject-specific, AU-informed audio-driven avatar system rather than as a fully open-domain talking-face generator.

Within the broader talking-head literature, HM-Talker can be read alongside diffusion-based motion-structured systems such as MoDiTalker, which separates talking-head generation into Audio-to-Motion (AToM) and Motion-to-Video (MToV) and emphasizes residual landmark diffusion, disentangled attention, and tri-plane-conditioned video synthesis (&&&^^^^5 OR ti:\5^^^^9&&&). The two papers belong to different model classes—^^^^5 OR abs:\5^^^^DGS-based hybrid motion fusion in HM-Talker versus diffusion-based motion-video disentanglement in MoDiTalker—but both treat explicit intermediate motion structure as preferable to direct audio-to-RGB mapping (&&&^^^^5search_query5^^^^&&&, &&&^^^^5 OR ti:\5^^^^9&&&).

6. Conceptual comparison and significance

The two HM-Talker systems address different modalities, yet they share a methodological preference for explicit structure in addition to sequence generation. In the speech-recognition HM-Talker, speaker-aware autoregressive decoding is supplemented by a speaker mask branch because timestamp tokens alone are described as insufficient for robust diarization in complex overlap (&&&^^^^5all:\5^^^^&&&). In the talking-head HM-Talker, implicit audio features are supplemented by explicit AUs because purely implicit audio-facial mapping is described as a source of motion blur, lip jitter, and phoneme-viseme misalignment (&&&^^^^5search_query5^^^^&&&).

This suggests a common design intuition: latent sequence models are useful for content generation, but difficult subproblems such as diarization or articulatory control benefit from dedicated explicit representations. In one case the explicit representation is a binary speaker activity mask; in the other it is an AU-based anatomical prior (&&&^^^^5all:\5^^^^&&&, &&&^^^^5search_query5^^^^&&&).

The two usages also occupy different positions relative to adjacent research. The speech HM-Talker is part of a line that includes SOT-based Whisper adaptation for multilingual meeting transcription (&&&^^^^5 OR abs:\5^^^^&&&), Sidecar- and TTI-based Whisper adaptation for joint multi-talker and target-talker ASR (&&&^^^^5 OR ti:\5^^^^&&&), and instruction-following speech-language modeling for cocktail-party transcription in MT-LLM (&&&^^^^5 OR ti:\5^^^^&&&). The talking-head HM-Talker belongs to a different trajectory that includes hybrid explicit/implicit motion modeling, ^^^^5 OR abs:\5^^^^D Gaussian Splatting renderers, and cross-modal motion control, with conceptual overlap with motion-disentangled video generation methods such as MoDiTalker (&&&^^^^5search_query5^^^^&&&, &&&^^^^5 OR ti:\5^^^^9&&&).

For encyclopedia purposes, the most precise interpretation is therefore terminological rather than genealogical: HM-Talker is a reused research name, not a unified benchmark or single algorithmic lineage. In speech literature it denotes a joint ASR-plus-diarization transformer centered on speaker masks (&&&^^^^5all:\5^^^^&&&). In talking-head synthesis it denotes a hybrid-motion, AU-guided ^^^^5 OR abs:\5^^^^DGS framework for high-fidelity lip-synchronized video generation (&&&^^^^5search_query5^^^^&&&).