Visual Speech Deduplication in LLM Pipelines

• The paper introduces a clustering-based deduplication method that collapses contiguous visual speech units to reduce sequence length by up to 47% without significant accuracy loss.
• It employs K-means clustering and averaging of latent features to compress input sequences, enhancing computational efficiency in visual speech recognition and translation.
• Integration with LoRA and LLM pipelines yields substantial FLOPs and memory savings while maintaining stable BLEU and WER performance.

A visual speech deduplication strategy is a principled approach to reducing computational redundancy in visual speech processing pipelines by compacting feature representations based on clustered temporal similarity. In the context of the VSP-LLM framework, deduplication operates by collapsing contiguous runs of video frames mapped to identical “visual speech units”—phoneme-like discrete representations of the input latent space—thereby compressing the sequence length input to a LLM without sacrificing recognition or translation accuracy. Empirical results demonstrate significant efficiency gains, with reductions in both the number of required floating-point operations (FLOPs) and memory consumption. This strategy integrates natively into pipelines leveraging Low-Rank Adaptation (LoRA), supporting scalable, context-aware visual speech recognition and translation (Yeo et al., 2024).

1. Extraction and Definition of Visual Speech Units

The deduplication process begins with each raw video frame $x_t$ ($t = 1, \ldots, T$) processed by a pre-trained self-supervised visual encoder $f_{ss}$ (specifically, AV-HuBERT). This mapping produces a latent vector $z_t = f_{ss}(x_t) \in \mathbb{R}^d$. All latent representations $z_t$ in the training data are pooled and subjected to $K$-means clustering, yielding $K$ centroids $\{c_1,\dots, c_K\}$, each denoted a visual speech unit.

During both training and inference, each frame’s latent $z_t$ is assigned a unit index $u_t$ via

$t = 1, \ldots, T$0

This procedure discretizes the sequence as $t = 1, \ldots, T$1, where $t = 1, \ldots, T$2 points to one of the $t = 1, \ldots, T$3 visual speech units.

2. Redundancy Criteria and Deduplication Algorithm

Redundancy is characterized by the equivalence of adjacent unit assignments. No explicit similarity threshold is used; redundancy exists wherever $t = 1, \ldots, T$4, so the binary indicator is

$t = 1, \ldots, T$5

Contiguous runs of identical unit indices are grouped: for a run $t = 1, \ldots, T$6 spanning frames $t = 1, \ldots, T$7 to $t = 1, \ldots, T$8, define $t = 1, \ldots, T$9. The deduplicated segment feature $f_{ss}$0 is the simple mean over corresponding latent vectors:

$f_{ss}$1

The deduplicated sequence $f_{ss}$2, where $f_{ss}$3, replaces the original per-frame latent sequence in subsequent LLM input formation.

3. Workflow and Pseudocode

The deduplication pipeline is described in the following pseudocode:

$z_t = f_{ss}(x_t) \in \mathbb{R}^d$6

These averaged features are then mapped to the LLM token-embedding space via a linear transformation and concatenated with natural language instructions before being consumed by the LLM.

4. Computational Benefits and Empirical Results

Deduplication produces a compressed latent sequence, empirically reducing the average sequence length on the MuAViC benchmark by approximately $f_{ss}$4 when using $f_{ss}$5 clusters ($f_{ss}$6). This compression yields substantial computational savings:

Setting	No Deduplication	With Deduplication (K=200)
FLOPs/train epoch	62.4 Peta	45.6 Peta (–26.9%)
FLOPs/inference	19.2 Peta	14.0 Peta (–27.1%)
Sequence length factor	1.00	0.53
Avg BLEU	14.6	14.5

No measurable loss in visual speech translation (BLEU: 14.6→14.5) or visual speech recognition (WER unchanged) was observed for $f_{ss}$7. Larger $f_{ss}$8 offer finer-grained units with less deduplication (higher fidelity, higher compute), while smaller $f_{ss}$9 lead to greater compression at the cost of marginal fidelity loss (e.g., $z_t = f_{ss}(x_t) \in \mathbb{R}^d$0 gives up to $z_t = f_{ss}(x_t) \in \mathbb{R}^d$1 FLOPs savings but minor BLEU drop to $z_t = f_{ss}(x_t) \in \mathbb{R}^d$2) (Yeo et al., 2024).

5. Integration with LLMs and LoRA

The deduplicated visual token sequence is linearly projected to match the LLM embedding space and combined with task instructions. The downstream LLM (LLaMA2-7B) is fine-tuned using QLoRA, updating only low-rank adapters. Since deduplication shortens the token sequence, each LoRA-adapted forward/backward pass processes fewer tokens, directly reducing per-step memory and compute requirements.

The number of clusters $z_t = f_{ss}(x_t) \in \mathbb{R}^d$3 governs the trade-off between computational savings and output fidelity. In practice, $z_t = f_{ss}(x_t) \in \mathbb{R}^d$4 provides an optimal trade-off for VSP-LLM, offering approximately 27% speedup with negligible BLEU or WER degradation.

6. Quantitative Benchmarks and Comparative Metrics

Experiments on MuAViC highlight the impact of deduplication within the VSP-LLM pipeline:

• Baseline (no deduplication): sequence length factor 1.00, FLOPs/train epoch 62.4 Peta, Avg BLEU 14.6.
• With deduplication ($z_t = f_{ss}(x_t) \in \mathbb{R}^d$5): sequence length factor 0.53, FLOPs/train epoch 45.6 Peta, Avg BLEU 14.5.
• Full VSP-LLM (dedup + LoRA): for 30 hours labeled training, VST BLEU = 18.2 versus 19.2 for a cascaded AV-HuBERT+MT pipeline trained with 433 hours. VSR WER is 29.8% (30 h data) and 26.7% (433 h data).

A summary of the key quantitative results appears below:

System	Training Data	VST BLEU	VSR WER	FLOPs/train epoch
Cascaded AV-HuBERT+MT	433 h	19.2	—	—
VSP-LLM (dedup+LoRA, K=200)	30 h	18.2	29.8%	45.6 Peta
VSP-LLM (dedup+LoRA, K=200)	433 h	—	26.7%	—

This suggests that deduplication is highly effective at compressing visual speech feature streams for LLM-based recognition and translation, with minimal loss in performance and significant computational efficiency gains.

7. Context, Significance, and Applicability

The deduplication strategy adopted in VSP-LLM introduces a lightweight, clustering-based compression mechanism tightly integrated with downstream LLM architectures and parameter-efficient adaptation (LoRA). Operating on the insight that contiguous frames with the same visual speech unit convey redundant information, deduplication exploits the temporal coherence of phoneme-like structures in visual speech data.

This approach is particularly significant for scalable, instruction-driven multi-task visual speech processing, making large multimodal models tractable for training and inference on long video sequences. It underscores the importance of temporal redundancy reduction when bridging raw modality streams with large, compute-hungry LLMs, without the need for hand-engineered similarity thresholds or segment boundaries. The method generalizes to visual speech recognition and translation, and is empirically validated as a robust, lossless compression technique in contemporary LLM pipelines (Yeo et al., 2024).