---
title: 'ViDove: Multimodal Video Translation'
url: https://www.emergentmind.com/topics/vidove
type: topic
---

# ViDove: Multimodal Video Translation

ViDove is an agent-based, retrieval-augmented translation system explicitly architected for multimodal, long-form video inputs. Leveraging audio, visual, and textual modalities in conjunction with two-tiered memory modules, ViDove aims to mirror the reasoning and context-integration behavior typical of expert human translators. Its pipeline jointly exploits agent orchestration, memory-augmented retrieval, domain-knowledge grounding, and multimodal fusion, resulting in significant improvements in video subtitle and general translation quality compared to state-of-the-art baselines [2507.07306].

## 1. System Architecture and Modular Workflow

ViDove decomposes translation into five interdependent modules, arranged in a pipelined architecture:

1. **Chunker / Timestamp Extractor**: Utilizes Pyannote-based speaker diarization to segment an input video $V$ into $k$ content chunks $C_1,\dots,C_k$, where each $C_i = (V_i, A_i)$ encodes the video frames and synchronized audio for a localized speaker turn.

2. **Auditory Agent $S$**: Performs automatic speech recognition (ASR) and paralinguistic cue extraction using Gemini-2.5-flash SpeechLM or Whisper-large-v3 over $A_i$. Output consists of a preliminary transcript and structured audio cues $cue_a$ (speaker, emotion, ambient sound), which are appended to short-term memory $M^s_a$ (a component of $M^s_{multimodal}$).

3. **Vision Agent $L^*$**: Extracts keyframes from $V_i$ and, together with visual short-term memory $M^s_{vision}$ and long-term domain memory $M^l_{domain}$, feeds these into a vision-language model (e.g., GPT-4o, Qwen2.5-VL) under a targeted analysis prompt $p_{analysis}$. The output is a semantic scene cue $cue_v$ and an updated visual memory state:
   $$
   cue_v = L^*(V_i, M^s_{vision}, M^l_{domain}, p_{analysis}).
   $$

4. **Translation Agent $L$**: Fuses transcript, $cue_a$, and $cue_v$ to form segment context $T_i = (cue_a, cue_v)$. Retrieval from $M^s$ yields recent translation history $H_i$ (past 5 source–target segment pairs) and current multimodal context $C_i$. Retrieval from $M^l_{domain}$ delivers a “domain guide” $D_i = query(M^l_{domain}, T_i)$. These are assembled into a translation prompt $p_{translation} = (base\text{-}prompt, H_i, C_i, D_i)$, passed to the translator agent $L_t$, e.g., GPT-4o:
   $$
   T_i^* = L_t(T_i, p_{translation}).
   $$

   A subsequent proofreader agent $L_{pr}$ suggests grammatical, terminological, and stylistic corrections using memory-attended input; finally, an editor agent $L_{ed}$ applies or rejects fixes, validates logical/temporal alignment, and emits the final subtitle segment $T_i^{ed}$. Updated pairs $(T_i,\,T_i^{ed})$ are stored in $M^s$.

5. **Post-Editing & Output Render**: A lightweight multi-agent post-processor enforces global subtitle consistency, merges contiguous segments, and outputs the finalized SRT file.

The translation process iterates over all $k$ chunks, maintaining dynamic memory states and ensuring multimodal, contextually-grounded translation at each segment.

## 2. Memory-Augmented Reasoning

ViDove’s architecture features a dual memory mechanism implemented using LLaMA-Index key–value stores:

- **Short-Term Memory ($M^s$):** Stores translation history (tuples of prior source and edited translations $(T_j, T_j^*)$) and most recent multimodal cues ($cue_a$, $cue_v$). 
- **Long-Term Memory ($M^l$):** Contains domain-specific resources such as glossaries, terminology tables, general-purpose web-derived documents, and expert guidance.

Memory access occurs through retrieval operations:
- **Read:** 
  - $H_i = retrieve(M^s_{history}, T_i)$
  - $C_i = retrieve(M^s_{context}, \cdot)$
  - $D_i = query(M^l_{domain}, T_i)$
- **Write:** 
  - $M^s_{history} \leftarrow M^s_{history} \cup \{(T_i, T_i^{ed})\}$

Retrieval ranks stored embedding keys $k_j$ against the current query using dot-product or cosine similarity, returning top-$k$ key–value pairs. There is no explicit soft attention rule, and all operations adhere to standard retrieval-augmented generation (RAG) semantics.

## 3. Multimodal Fusion and Context Integration

Fusion is central to both audio and text processing:

- **ASR Enhancement:** The auditory agent $S$ incorporates stored visual cues ($M^s_a, M^s_v$) as auxiliary context, injecting discriminative keywords or resolving ambiguous pronunciations via SpeechLM’s input.
- **Translation Prompt Construction:** The translation agent concatenates the fused context vector $(cue_a \oplus cue_v)$ with translation history $H_i$ and domain guide $D_i$. Formally:
  $$
  T_i^* = L_t\left( T_i; \left[ cue_a \oplus cue_v \oplus H_i \oplus D_i \right] \right),
  $$
  where $\oplus$ denotes prompt concatenation within the LLM.

This unified fusion enables text generation to be directly conditioned on synchronously observed acoustic and visual semantics, as well as explicit memory-driven context retrieval.

## 4. Domain Knowledge and Retrieval Integration

ViDove’s long-term memory $M^l_{domain}$ is pre-populated offline by ingesting formal glossaries, human-generated domain notes, and project-specific documents. At inference time, per-segment queries retrieve a “domain guide” – such as paraphrase samples, targeted terminology, or subject-relevant explanatory text – that is appended to the translation prompt.

No LLM or model parameters are re-trained for domain adaptation; all external knowledge integration occurs via context injection through retrieval-augmented prompting. This design enables rapid adaptation to new domains and vocabulary sets without additional supervised fine-tuning.

## 5. Evaluation, Ablation, and Benchmarking Results

ViDove is evaluated on DoveBench, a purpose-built benchmark comprising 50 long-form videos (4.97 hours Counter-Strike, 12.27 hours StarCraft II) with 16,968 manually aligned and timestamped Chinese subtitles. Metrics include BLEU, SubER (subtitle edit distance plus shift), BLEURT, and SubSONAR. Key results on DoveBench:

| System                | BLEU ↑ | SubER ↓ | BLEURT ↑ | SubSONAR ↑ |
|-----------------------|--------|---------|----------|------------|
| ViDove                | 23.51  | 73.38   | 19.55    | 0.39       |
| Whisper + DelTA (SoTA)| 18.26  | 86.83   | 12.30    | 0.28       |

Relative gains: +28.8% BLEU, −15.5% SubER, +58.9% BLEURT, highest SubSONAR.

On the BigVideo dataset (9,981 h, 4.5M pairs), ViDove remains competitive (BLEU = 26.05 vs. 29.09, sCOMET = 0.73 vs. 0.75).

Ablation on a SC2 subset of DoveBench demonstrates the critical importance of each system component:

- Removing domain memory: BLEU drops to 14.86 (−6.98), SubER rises to 77.31
- Without visual cues: BLEU 14.56, SubER 77.55
- Without proofreader: BLEU 13.56, SubER 80.76

The largest single drop occurs upon removing the proofreader agent, indicating the significance of multi-agent post-editing for translation quality.

## 6. Implementation and Technical Configuration

ViDove’s components and technological stack are as follows:

- **Auditory Agent:** Gemini-2.5-flash SpeechLM and Whisper-large-v3 ASR backends.
- **Visual and Translation Agents:** GPT-4o or equivalent large vision–language models.
- **Memory Engine:** LLaMA-Index, utilizing cosine similarity retrieval for all key–value memory operations.
- **Chunking:** Pyannote-based speaker diarization for temporal segmentation.
- **Agent Orchestration:** All large language models operate in strict zero-shot or few-shot mode; no additional supervised training or parameter fine-tuning.
- **Prompt Templates:** Custom-designed for granular line alignment, style, and targeted domain instructions.

Output is finalized through a post-processing step that promotes output consistency, merges adjacent subtitles, and ensures proper SRT formatting.

## 7. Significance and Implications for Multimodal Machine Translation

ViDove demonstrates that orchestrated, agent-based pipelines, equipped with retrieval-augmented memory and multimodal fusion, can deliver marked improvements in long-form multimodal translation tasks—without the need for model retraining or end-to-end fine-tuning. The design leverages prompt engineering, effective context retrieval, and multi-stage post-editing, achieving context carry-over and terminological fidelity on extended video material. The introduction of DoveBench as an evaluation resource enables systematic benchmarking for future research on multimodal subtitle translation under realistic, high-complexity constraints [2507.07306].

Source: https://www.emergentmind.com/topics/vidove