MMTM: Tri-Modal Topic Modeling for Long-Form Video via Similarity-Gated Fusion

Abstract: We introduce MMTM, a modular pipeline for topic discovery in long-form video that integrates speech recognition, audio and visual embeddings, and BERTopic clustering through a deterministic similarity-gated fusion. Evaluated cross-lingually on German (Tagesschau) and English (NBC) broadcast news, joint tri-modal modeling substantially improves topic quality: noise drops from 0.27 to 0.06, transition rate from 0.70 to 0.21, and normalized entropy rises from 0.84 to 0.92, indicating more coherent and temporally stable topics. Cluster validity (Calinski-Harabasz) improves by 5-12X across embedding spaces. Lexical coherence (NPMI) rises from 0.77 to 0.86 on German but is corpus-dependent and does not transfer to the shorter NBC broadcasts. We release the pipeline code and a human-validated 54-hour multimodal video topic corpus with dual-annotator visual evaluation and LLM-assisted labeling.

• The paper introduces a tri-modal fusion pipeline that deterministically integrates ASR, audio, and visual representations for enhanced topic discovery.
• It demonstrates significant improvements in segmentation coherence and cluster geometry through extensive cross-lingual and multimodal evaluations.
• Ablation studies reveal the pivotal role of visual embeddings and the need for adaptive gating to effectively balance modality contributions.

Tri-Modal Topic Modeling for Long-Form Video: The MMTM Pipeline

Introduction

The paper "MMTM: Tri-Modal Topic Modeling for Long-Form Video via Similarity-Gated Fusion" (2605.29765) presents a modular, deterministic pipeline for topic discovery in long-form video. The approach integrates segment-level representations from text (ASR transcripts), audio (CLAP/MFCC/Pyannote), and visual (OpenCLIP/SigLIP/Qwen3-VL) streams, fusing these via a parameter-free similarity-gated mechanism and clustering with BERTopic. The contributions include a cross-lingual evaluation on German and English broadcast news corpora, a human-validated 54-hour annotated dataset, and extensive ablations on the impact of encoder and modality selection. MMTM demonstrates substantial improvements in topic segmentation coherence, cluster geometry, and cross-modal semantic alignment compared to unimodal and bimodal baselines.

MMTM Architecture and Pipeline

MMTM operates as a modular pipeline, partitioned into ASR transcription (Whisper), acoustic and visual embedding extraction, deterministic gating-based fusion, and topic clustering. The multimodal fusion is performed at the segment level, with segment definitions following lexical ASR boundaries rather than fixed windowing, increasing semantic consistency. Each segment is represented by three parallel streams:

• Text: Whisper-based ASR transcripts, post-processed via frequency-based lexical filtering.
• Audio: CLAP-based embeddings encoding non-lexical acoustic content, with fallback on MFCC/Pyannote for robustness.
• Visual: OpenCLIP/SigLIP/VL-Large embeddings from a diversity-ranked frame selection process, yielding high-fidelity, non-redundant segment visual summaries.

The core innovation is the similarity-gated fusion: all segment embeddings are L2-normalized and truncated to the minimum common dimensionality, then pairwise and triple-wise similarity scores are computed and used to reweight modalities. The fusion vector includes weighted modality embeddings and their Hadamard interactions, finalized with a global L2 normalization prior to BERTopic clustering.

Figure 1: The architecture of MMTM, integrating modular ASR, audio embedding, visual embedding, and deterministic similarity-gated fusion prior to BERTopic topic clustering.

Evaluation: Structure, Geometry, and Coherence

MMTM is evaluated primarily on two corpora: a 54-hour daily German Tagesschau news set and a 20-hour NBC English news collection, providing strong cross-lingual and cross-format coverage. The main structural metrics (noise, transition rate, topic entropy, and topic count) and geometric scores (Calinski-Harabasz (CH), Silhouette, Davies-Bouldin) are reported across unimodal, bimodal, and tri-modal configurations. Multimodal fusion yields consistent, substantial improvements:

• Structural Coherence: For Tagesschau, noise drops from 0.27 to 0.06, transition rate from 0.70 to 0.21, and entropy climbs from 0.84 to 0.92. Similar patterns are observed on NBC, although lexical coherence gains are attenuated on the shorter English segments.
• Cluster Geometry: Visual-space CH increases by up to 12x (e.g., from 2.81 to 31.89), with Silhouette scores flipping from negative (text-only) to solidly positive (>0.36) under multimodal assignment.
• Semantic Coherence: Lexical NPMI increases from 0.77 to 0.86 (Tagesschau), indicating tighter and more interpretable topic clusters; image embedding coherence (IEC) in fused space rises from 0.59 to 0.79.
• Modality Dominance: Ablations show that visual embeddings drive the bulk of the improvement, while audio must be gated to avoid overwhelming lexical/semantic evidence.

  Figure 2: Relative improvement (\%) of multimodal encoders over text-only baseline across primary topic structure and coherence metrics.

Fusion Ablations and Encoder Selection

Systematic encoder ablations demonstrate the robustness of the fusion paradigm and surface key trade-offs:

• Visual Backends: OpenCLIP balances structural (noise, transition) and lexical (NPMI) quality; SigLIP yields the highest cross-modal alignment (IEC fused = 0.808); Qwen3-VL, although large, underperforms in cross-modal topic coherence, highlighting the importance of alignment rather than scale.
• Audio Backends: CLAP offers stronger semantic alignment, while Pyannote achieves cleaner speaker-like separation but only moderate NPMI gains. This evidences a trade-off between acoustic scene coverage and speaker ID focus.
• Gate Limitation: The used deterministic gate, being parameter-free, cannot adapt weights at inference; this is effective due to strong prior training but precludes context-sensitive reweighting, as evidenced by flat performance in visually monotonous studio segments.

Human Validation

A dual-stage human annotation protocol—LLM-assisted topic labeling and dual-annotator visual matching/intrusion tests—confirms the interpretability and semantic coherence of discovered topics. Annotator results mirror automated metrics: visually distinctive topics (e.g., weather, sports, conflict) yield high agreement and coherence ratings; visually ambiguous studio content remains more challenging.

Theoretical and Practical Implications

The strong tri-modal improvements in topic stability, cluster consistency, and cross-modal alignment suggest that dense multimodal fusion is critical for topical analysis in long-form, real-world media, where text alone is insufficient and naive concatenation can degrade results. The work positions MMTM as a reproducible reference pipeline, offering:

• Benchmark Corpus and Tooling: Release of a human-validated, large-scale, multimodal video-topic corpus; open-source modular pipeline; annotation toolkit.
• Research Directions: The current similarity-gated, parameter-free fusion highlights the limitations of static weightings—future models should explore adaptive, uncertainty- or content-aware gates and extend to non-broadcast, less-structured media.

Conclusion

MMTM provides the first deterministic, modular, reproducible pipeline for tri-modal topic segmentation of long-form video, integrating ASR, audio, and visual evidence and producing state-of-the-art topic coherence, stability, and cross-modal cluster geometry. While visual streams provide dominant structure, prudent gating and joint fusion are essential: fusion not only projects lexical topics onto visual clusters but yields genuinely coherent, shared topic partitions across modalities. Limitations remain regarding ASR error propagation, static modality weighting, and generalization beyond the news/video domain. Overall, the methodology establishes a practical and replicable standard for future multimodal topic modeling pipelines, with released code and benchmarks supporting community progress in this domain.