ViDove: Multimodal Video Translation

• ViDove is a multimodal translation system that leverages agent orchestration, memory-augmented retrieval, and fusion of audio, visual, and textual data.
• It decomposes video translation into five interdependent modules, covering speaker diarization, auditory and visual analysis, and dynamic memory-enhanced translation prompting.
• Evaluation on DoveBench shows significant improvements over state-of-the-art methods, underscoring the critical role of multi-agent post-editing and domain-specific knowledge retrieval.

ViDove is an agent-based, retrieval-augmented translation system explicitly architected for multimodal, long-form video inputs. Leveraging audio, visual, and textual modalities in conjunction with two-tiered memory modules, ViDove aims to mirror the reasoning and context-integration behavior typical of expert human translators. Its pipeline jointly exploits agent orchestration, memory-augmented retrieval, domain-knowledge grounding, and multimodal fusion, resulting in significant improvements in video subtitle and general translation quality compared to state-of-the-art baselines (Lu et al., 9 Jul 2025).

1. System Architecture and Modular Workflow

ViDove decomposes translation into five interdependent modules, arranged in a pipelined architecture:

1. Chunker / Timestamp Extractor: Utilizes Pyannote-based speaker diarization to segment an input video $V$ into $k$ content chunks $C_1,\dots,C_k$, where each $C_i = (V_i, A_i)$ encodes the video frames and synchronized audio for a localized speaker turn.
2. Auditory Agent $S$: Performs automatic speech recognition (ASR) and paralinguistic cue extraction using Gemini-2.5-flash SpeechLM or Whisper-large-v3 over $A_i$. Output consists of a preliminary transcript and structured audio cues $cue_a$ (speaker, emotion, ambient sound), which are appended to short-term memory $M^s_a$ (a component of $M^s_{multimodal}$).
3. Vision Agent $L^*$: Extracts keyframes from $k$0 and, together with visual short-term memory $k$1 and long-term domain memory $k$2, feeds these into a vision-LLM (e.g., GPT-4o, Qwen2.5-VL) under a targeted analysis prompt $k$3. The output is a semantic scene cue $k$4 and an updated visual memory state:

$k$5

1. Translation Agent $k$6: Fuses transcript, $k$7, and $k$8 to form segment context $k$9. Retrieval from $C_1,\dots,C_k$0 yields recent translation history $C_1,\dots,C_k$1 (past 5 source–target segment pairs) and current multimodal context $C_1,\dots,C_k$2. Retrieval from $C_1,\dots,C_k$3 delivers a “domain guide” $C_1,\dots,C_k$4. These are assembled into a translation prompt $C_1,\dots,C_k$5, passed to the translator agent $C_1,\dots,C_k$6, e.g., GPT-4o:

$C_1,\dots,C_k$7

A subsequent proofreader agent $C_1,\dots,C_k$8 suggests grammatical, terminological, and stylistic corrections using memory-attended input; finally, an editor agent $C_1,\dots,C_k$9 applies or rejects fixes, validates logical/temporal alignment, and emits the final subtitle segment $C_i = (V_i, A_i)$0. Updated pairs $C_i = (V_i, A_i)$1 are stored in $C_i = (V_i, A_i)$2.

2. Post-Editing & Output Render: A lightweight multi-agent post-processor enforces global subtitle consistency, merges contiguous segments, and outputs the finalized SRT file.

The translation process iterates over all $C_i = (V_i, A_i)$3 chunks, maintaining dynamic memory states and ensuring multimodal, contextually-grounded translation at each segment.

2. Memory-Augmented Reasoning

ViDove’s architecture features a dual memory mechanism implemented using LLaMA-Index key–value stores:

• Short-Term Memory ($C_i = (V_i, A_i)$4): Stores translation history (tuples of prior source and edited translations $C_i = (V_i, A_i)$5) and most recent multimodal cues ($C_i = (V_i, A_i)$6, $C_i = (V_i, A_i)$7).
• Long-Term Memory ($C_i = (V_i, A_i)$8): Contains domain-specific resources such as glossaries, terminology tables, general-purpose web-derived documents, and expert guidance.

Memory access occurs through retrieval operations:

• Read:
  • $C_i = (V_i, A_i)$9
  • $S$0
  • $S$1
• Write:
  • $S$2

Retrieval ranks stored embedding keys $S$3 against the current query using dot-product or cosine similarity, returning top-$S$4 key–value pairs. There is no explicit soft attention rule, and all operations adhere to standard retrieval-augmented generation (RAG) semantics.

3. Multimodal Fusion and Context Integration

Fusion is central to both audio and text processing:

• ASR Enhancement: The auditory agent $S$5 incorporates stored visual cues ($S$6) as auxiliary context, injecting discriminative keywords or resolving ambiguous pronunciations via SpeechLM’s input.
• Translation Prompt Construction: The translation agent concatenates the fused context vector $S$7 with translation history $S$8 and domain guide $S$9. Formally:

$A_i$0

where $A_i$1 denotes prompt concatenation within the LLM.

This unified fusion enables text generation to be directly conditioned on synchronously observed acoustic and visual semantics, as well as explicit memory-driven context retrieval.

4. Domain Knowledge and Retrieval Integration

ViDove’s long-term memory $A_i$2 is pre-populated offline by ingesting formal glossaries, human-generated domain notes, and project-specific documents. At inference time, per-segment queries retrieve a “domain guide” – such as paraphrase samples, targeted terminology, or subject-relevant explanatory text – that is appended to the translation prompt.

No LLM or model parameters are re-trained for domain adaptation; all external knowledge integration occurs via context injection through retrieval-augmented prompting. This design enables rapid adaptation to new domains and vocabulary sets without additional supervised fine-tuning.

5. Evaluation, Ablation, and Benchmarking Results

ViDove is evaluated on DoveBench, a purpose-built benchmark comprising 50 long-form videos (4.97 hours Counter-Strike, 12.27 hours StarCraft II) with 16,968 manually aligned and timestamped Chinese subtitles. Metrics include BLEU, SubER (subtitle edit distance plus shift), BLEURT, and SubSONAR. Key results on DoveBench:

System	BLEU ↑	SubER ↓	BLEURT ↑	SubSONAR ↑
ViDove	23.51	73.38	19.55	0.39
Whisper + DelTA (SoTA)	18.26	86.83	12.30	0.28

Relative gains: +28.8% BLEU, −15.5% SubER, +58.9% BLEURT, highest SubSONAR.

On the BigVideo dataset (9,981 h, 4.5M pairs), ViDove remains competitive (BLEU = 26.05 vs. 29.09, sCOMET = 0.73 vs. 0.75).

Ablation on a SC2 subset of DoveBench demonstrates the critical importance of each system component:

• Removing domain memory: BLEU drops to 14.86 (−6.98), SubER rises to 77.31
• Without visual cues: BLEU 14.56, SubER 77.55
• Without proofreader: BLEU 13.56, SubER 80.76

The largest single drop occurs upon removing the proofreader agent, indicating the significance of multi-agent post-editing for translation quality.

6. Implementation and Technical Configuration

ViDove’s components and technological stack are as follows:

• Auditory Agent: Gemini-2.5-flash SpeechLM and Whisper-large-v3 ASR backends.
• Visual and Translation Agents: GPT-4o or equivalent large vision–LLMs.
• Memory Engine: LLaMA-Index, utilizing cosine similarity retrieval for all key–value memory operations.
• Chunking: Pyannote-based speaker diarization for temporal segmentation.
• Agent Orchestration: All LLMs operate in strict zero-shot or few-shot mode; no additional supervised training or parameter fine-tuning.
• Prompt Templates: Custom-designed for granular line alignment, style, and targeted domain instructions.

Output is finalized through a post-processing step that promotes output consistency, merges adjacent subtitles, and ensures proper SRT formatting.

7. Significance and Implications for Multimodal Machine Translation

ViDove demonstrates that orchestrated, agent-based pipelines, equipped with retrieval-augmented memory and multimodal fusion, can deliver marked improvements in long-form multimodal translation tasks—without the need for model retraining or end-to-end fine-tuning. The design leverages prompt engineering, effective context retrieval, and multi-stage post-editing, achieving context carry-over and terminological fidelity on extended video material. The introduction of DoveBench as an evaluation resource enables systematic benchmarking for future research on multimodal subtitle translation under realistic, high-complexity constraints (Lu et al., 9 Jul 2025).