WikiVideo: Multi-Video Article Generation

• WikiVideo is a benchmark for synthesizing Wikipedia-style articles from multiple event videos by grounding claims in dense, multimodal evidence.
• The framework employs an iterative Collaborative Article Generation (CAG) process that interleaves video summarization with reasoning to focus on high-level event semantics.
• It tackles challenges in retrieval and multimodal fusion, offering actionable insights into video-based retrieval-augmented generation methods.

WikiVideo is a benchmark and task for automatically creating a high-level Wikipedia-style article that aggregates information from multiple diverse videos about real-world events such as natural disasters or political elections. It was introduced to close a gap between retrieval-augmented generation workflows, which focus heavily on text, and existing video-based summarization methods, which focus on low-level scene understanding rather than high-level event semantics. The benchmark couples expert-written articles with densely annotated videos that provide evidence for articles’ claims, and it is paired with Collaborative Article Generation (CAG), an interactive method that interleaves a VideoLLM with an r1-style reasoning model to produce more focused event-level summaries than VideoLLMs alone (Martin et al., 1 Apr 2025).

1. Problem definition and scope

WikiVideo defines multi-video article generation as a grounded generation problem over event-centric multimodal evidence. Each example $i$ is a tuple $(T_i, Q_i, V_i, A_i^*)$, where $T_i$ is the event topic, $Q_i$ is a short query, often simply the event name, $V_i=\{v_{i1},\dots,v_{in}\}$ is the set of relevant videos, and $A_i^*$ is the expert-written reference article lead. The model with parameters $\theta$ must produce a lead $A_i$ that maximizes the conditional log-likelihood over the reference tokens $y_{i1\dots iM}$ (Martin et al., 1 Apr 2025).

The task objective is given in the usual autoregressive form:

$$L(\theta)=-\sum_{i=1}^N \sum_{j=1}^{|A_i^*|} \log P_\theta(y_{ij}\mid y_{i,<j},V_i,Q_i).$$

Here, $(T_i, Q_i, V_i, A_i^*)$0 is implemented by a retrieval-augmented transformer that digests video summaries, and optionally transcripts, as context (Martin et al., 1 Apr 2025).

The problem setting is motivated by two observations. First, videos are intuitive sources for retrieval-augmented generation because they often contain direct evidence about real-world events. Second, off-the-shelf VideoLLMs tend to fixate on low-level visual features and low-level scene descriptions rather than high-level event semantics. WikiVideo therefore frames article generation not as generic summarization, but as evidence-grounded synthesis over multiple heterogeneous video sources (Martin et al., 1 Apr 2025).

2. Benchmark construction and annotation design

WikiVideo is built on top of the MultiVENT video-event corpora and English Wikipedia. It comprises 52 real-world event topics, supported by a total of 398 “relevant” videos, drawn from a larger 109 K-video lake used for retrieval-augmented generation experiments. On average there are $(T_i, Q_i, V_i, A_i^*)$1 videos per event with $(T_i, Q_i, V_i, A_i^*)$2, and each video runs $(T_i, Q_i, V_i, A_i^*)$3 s in duration, with minimum $(T_i, Q_i, V_i, A_i^*)$4 s and maximum $(T_i, Q_i, V_i, A_i^*)$5 s (Martin et al., 1 Apr 2025).

Dataset creation follows a five-stage annotation pipeline. In Initial Event Selection, events from MultiVENT 1.0/2.0 linked to English Wikipedia articles are selected, yielding 58 candidates. In Claim Decomposition, Wikipedia lead sentences are automatically split into atomic “subclaims” by few-shot prompting of Qwen 2.5 32B, then manually corrected by experts so that each subclaim is decontextualized and verifiable. In Claim Grounding, expert annotators watch each video and mark, for every subclaim, whether it is supported by the video’s visual content, on-screen text, audio, or none. In Article Rewriting, the original lead is rewritten so that it contains exactly the grounded subclaims and no ungrounded material, yielding 52 fully grounded reference articles (Martin et al., 1 Apr 2025).

The benchmark is densely annotated at the claim level. Per-topic averages include 51.1 total subclaims, with $(T_i, Q_i, V_i, A_i^*)$6, $(T_i, Q_i, V_i, A_i^*)$7, and $(T_i, Q_i, V_i, A_i^*)$8. A pilot estimated Krippendorff’s $(T_i, Q_i, V_i, A_i^*)$9 overall for claim grounding, with video $T_i$0, audio $T_i$1, and OCR $T_i$2 (Martin et al., 1 Apr 2025).

These design choices make WikiVideo a grounded benchmark rather than a loosely aligned collection of videos and summaries. The article leads are constrained to grounded evidence, and the annotations separate support from frames, audio, and OCR. That structure is central to its stated role as the first large-scale, densely grounded benchmark for multi-video article generation (Martin et al., 1 Apr 2025).

3. Collaborative Article Generation

Collaborative Article Generation is the method proposed for article creation from multiple videos. It is designed to address two hurdles: off-the-shelf VideoLLMs focus on low-level scene descriptions rather than high-level event semantics, and processing multiple long videos at once is memory-prohibitive. CAG therefore interleaves a VideoLLM with a text-based “reasoner” in a relevance-feedback loop, then hands all collected summaries to a text-only LLM for final synthesis (Martin et al., 1 Apr 2025).

The architecture has three roles. A VideoLLM, such as QwenVL2.5-72B, InternVideo2.5, or LLaVA-Video, takes a video $T_i$3 and a prompt $T_i$4 and returns a text summary $T_i$5. A reasoning model, specifically DeepSeek-R1 distilled to Qwen-32B, examines the current summary $T_i$6 and the query $T_i$7. If the summary is already sufficiently focused on the target event, it outputs “no new query”; otherwise, it crafts a new prompt $T_i$8 that requests missing event-relevant details. A text-only LLM then synthesizes the final article from the accumulated summaries (Martin et al., 1 Apr 2025).

The per-video loop is iterative. It begins with a generic prompt such as “Describe the video in detail,” then refines the prompt through up to a fixed iteration budget $T_i$9. In pseudocode form:

$V_i=\{v_{i1},\dots,v_{in}\}$3

The paper does not introduce a novel transformer mechanism beyond existing VideoLLM and LLM training; within the transformers, standard scaled-dot-product attention is used:

$Q_i$0

CAG is thus a coordination strategy rather than a new pretraining regime. Its central claim is procedural: iterative relevance feedback between a VideoLLM and a reasoning model can elicit more focused, higher-level event summaries than direct one-pass video description alone (Martin et al., 1 Apr 2025).

4. Retrieval settings and multimodal evidence flow

WikiVideo supports two retrieval settings. In Oracle, $Q_i$1 is exactly the set of gold-annotated relevant videos for event $Q_i$2, with an average of 7.65 videos per event. In RAG, $Q_i$3 is obtained by ranking all 109 K test-set videos using a retrieval model and selecting the top-5. The retrieved videos are then passed through the same CAG or baseline generation pipeline to produce the article $Q_i$4 (Martin et al., 1 Apr 2025).

Two retrievers are evaluated. Video-ColBERT is a video-only late-interaction bi-encoder that learns video and text embeddings and ranks by cosine similarity; it achieves $Q_i$5. MMMORRF is an audio-visual retriever that fuses frame, OCR, and transcript features; it achieves $Q_i$6 (Martin et al., 1 Apr 2025).

This retrieval formulation is important because WikiVideo is not limited to a closed set of already identified evidence videos. It also functions as a testbed for end-to-end multimodal retrieval-augmented generation, where the system must first locate useful videos in a much larger video lake and then synthesize a grounded article from them. A plausible implication is that the benchmark simultaneously probes retrieval quality, video understanding, and long-form synthesis rather than isolating any one of those components.

5. Evaluation protocol and empirical results

Evaluation combines overlap-based, embedding-based, and content-structure metrics. Automatic metrics include ROUGE-1/2/L F1 (R1, R2, RL) for n-gram and longest-common-subsequence overlap, BERTScore F1 (BS) for token-embedding similarity, AlignScore (AS) for factual consistency as a learned text-pair alignment score in $Q_i$7, and Argument F1 (Arg), which uses GPT-4o to answer event-type role questions such as “Who won? When was election?” on both prediction and reference and then computes maximum-matching F1 over extracted spans. Human evaluation is limited: the reported human statistic is claim-grounding inter-annotator agreement, with Krippendorff $Q_i$8 overall, and there is no end-to-end human article evaluation beyond comparison to Wikipedia and a small crowd-sourced annotation for upper bound (Martin et al., 1 Apr 2025).

In oracle, vision-only experiments, naïve Concat baselines that concatenate generic VideoLLM summaries or only their RePrompted versions yield very low R1, approximately 7–16. Adding a text-only aggregator LLM with zero RePrompting, denoted CAG-0, raises R1 to 19–33 depending on the VideoLLM. Enabling only $Q_i$9 RePrompting iterations, the full CAG configuration further boosts R1 to 33.9 with InternVideo2.5 and 33.96 with QwenVL2.5, a relative improvement of +1–3 points over CAG-0. Argument F1 similarly climbs from approximately 20 to approximately 30 (Martin et al., 1 Apr 2025).

A common assumption that transcript concatenation straightforwardly improves multimodal generation is not supported by the reported results. Adding audio transcripts slightly degrades performance across the board; for example, with QwenVL2.5, CAG R1 falls from 33.96 to 32.05. The paper attributes this possibly to VideoLLMs not being pretrained on transcripts and to the aggregator under-utilizing audio (Martin et al., 1 Apr 2025).

Under RAG, the performance loss is substantial. CAG’s R1 drops from 33.96 in oracle to 24.13 with Video-ColBERT plus QwenVL2.5, or to approximately 23.8–24.1 with MMMORRF. Argument F1 likewise falls from approximately 30.8 to approximately 20.7. The stated interpretation is that retrieval errors, especially irrelevant videos, lead the aggregator to dilute focus (Martin et al., 1 Apr 2025).

No formal statistical tests are reported, but the paper notes consistent gains of more than 1 R1 point across seeds and VideoLLMs. Error analyses show that CAG still hallucinates details not grounded in video, such as political context, and struggles with under-represented event types, including natural disasters and demonstrations, where visual content is noisier (Martin et al., 1 Apr 2025).

6. Relationship to short-video encyclopedia systems

WikiVideo should be distinguished from short-video encyclopedia systems that organize videos as structured knowledge assets rather than synthesizing a single article from multiple event videos. Kuaipedia is the clearest contrast in the supplied literature: it is a large-scale multi-modal encyclopedia consisting of items, aspects, and short videos linked to them, extracted from billions of videos of Kuaishou. Its core structure is an Item–Aspect Tree and a collection of linked item–aspect–video triplets rather than an event-level article-generation benchmark (Pan et al., 2022).

The scale of Kuaipedia is much larger in item coverage: more than 26 million items, more than 2.5 million aspects, approximately 70 million candidate item–aspect pairs, more than 200 million knowledge videos, and approximately 100 million linked item–aspect video triplets. It defines a new task, multi-modal item-aspect linking, in which a video is grounded to an $V_i=\{v_{i1},\dots,v_{in}\}$0 pair using a two-phase model with recall and ranking. Reported intrinsic results include Top-1 video$V_i=\{v_{i1},\dots,v_{in}\}$1item accuracy of 90.0% and Top-1 video$V_i=\{v_{i1},\dots,v_{in}\}$2(item, aspect) accuracy of 82.8%, while extrinsic experiments show gains on entity typing and entity linking when Kuaipedia aspects and video embeddings are appended to the input (Pan et al., 2022).

The contrast clarifies what is specific about WikiVideo. Kuaipedia is organized around broad item knowledge and aspect browsing; WikiVideo is organized around event-centric, claim-grounded, multi-video article generation. This suggests two complementary paradigms for multimodal encyclopedic knowledge: a graph-centric paradigm built from item–aspect–video relations, and an overview-centric paradigm built from grounded event articles.

7. Limitations, applications, and research directions

Several limitations are explicit. Audio integration remains weak: audio transcripts shortened the generated leads and hurt coverage, and better multimodal fusion rather than simple transcript concatenation is identified as a next step. VideoLLMs are not trained for high-level semantics: fine-tuning on event-summary objectives such as WikiVideo may close the gap. Inference-time cost is high because the CAG loop relies on off-the-shelf reasoning and LLMs, while joint training of the three components remains unexplored. Retrieval errors in RAG drastically reduce performance, making stronger video retrievers and end-to-end RAG fine-tuning promising directions (Martin et al., 1 Apr 2025).

The benchmark also exposes a narrower but consequential limitation in evaluation practice. Since there is no end-to-end human article evaluation beyond comparison to Wikipedia and a small crowd-sourced annotation for upper bound, the reported gains are primarily metric-based. This does not negate the results, but it constrains how directly they can be interpreted as improvements in encyclopedic writing quality.

The applications proposed for WikiVideo are operational rather than speculative. They include rapid generation of Wikipedia-style overviews for breaking news events using citizen-generated footage, journalist assistants that synthesize amateur and professional clips into coherent reports, and multimedia fact-checking tools that tie each written claim back to specific video evidence (Martin et al., 1 Apr 2025).

Taken together, WikiVideo and CAG establish a technically specific research program: integrating video into retrieval-augmented generation through densely grounded event benchmarks, iterative relevance feedback, and long-form synthesis from multiple videos. The benchmark’s strongest contribution is not only the article-generation task itself, but also the explicit exposure of unresolved problems in audio-visual fusion, event-level semantic abstraction, and multimodal retrieval (Martin et al., 1 Apr 2025).