PlotTree: Hierarchical Reasoning for Long Videos

• PlotTree is a hierarchical video understanding agent that organizes long-form videos into multi-level plot nodes for improved narrative reasoning.
• It leverages pretrained modules for face recognition, captioning, and summarization to preserve local actions and global storyline structure.
• Empirical results on StoryVideoQA confirm that PlotTree outperforms flat retrieval models, excelling in long-range dependency and inference tasks.

Searching arXiv for the exact "PlotTree" paper and closely related benchmark context. PlotTree is a video understanding agent for deep video understanding (DVU) in long-form story videos. It was introduced alongside the StoryVideoQA benchmark as a method that re-organizes long-range video content into a hierarchical plot structure, so that question answering can proceed through retrieval and reasoning over plot nodes rather than over a flat set of captions or raw clips. In this usage, PlotTree denotes a hierarchical multimodal reasoning system rather than a generic tree-visualization utility. Its immediate context is long-video question answering over TV episodes and movies, where existing methods were found to struggle with long-range temporal dependency, instance-level story elements, and inference-heavy reasoning (Wu et al., 4 Jun 2026).

1. Problem setting and scope

PlotTree was proposed for the StoryVideoQA setting, where the underlying task is not short-clip factoid VideoQA but DVU over story videos. The benchmark contains over 363K multiple-choice QA pairs on 393.2 hours of videos, including TV series with average duration 1,635 seconds and movies with average duration 7,878 seconds. It is balanced across 14 fine-grained topics formed by two question types—perception and inference—and seven story-element combinations $C, A, L, CA, CL, AL, CAL$. The associated study reports that state-of-the-art VideoQA methods “cannot fully maintain long-range character associations and construct a coherent understanding of storylines,” which is the failure mode PlotTree is designed to address (Wu et al., 4 Jun 2026).

PlotTree is downstream of StoryMindv2 rather than part of StoryMindv2’s QA-generation pipeline. StoryMindv2 constructs the benchmark and associated resources, including time-aligned subtitles and a character library. PlotTree consumes those resources during inference. This division of labor is important: the dataset pipeline provides multimodal story structure, while PlotTree provides a reasoning architecture specialized for long-range narrative comprehension.

The motivating claim is that flat retrieval over captions is misaligned with DVU. Story understanding is hierarchical: local actions and dialogue must be related to scenes, scenes to subplots, and subplots to a coherent global storyline. PlotTree operationalizes that premise by building a hierarchy whose leaves are grounded local plot units and whose upper levels are progressively condensed plot summaries.

2. Inputs, components, and system organization

PlotTree uses four main inputs: sampled video frames, subtitles or dialogue aligned to those frames, a character library containing portrait images, and a question $q$. In the reported implementation, frames are sampled at 1 fps. The system uses InsightFace for face recognition, LLaVA-1.6 for plot captioning, Qwen3 embeddings for semantic retrieval and clustering, and Gemini-2.0-Flash for plot condensation and final QA reasoning; stronger-agent experiments replace that backend with Gemini-3-Flash (Wu et al., 4 Jun 2026).

The architecture has two phases. The first is PlotTree Construction, which creates a reusable hierarchy for a video. The second is PlotTree QA, which retrieves relevant nodes from that hierarchy and answers a question. The paper emphasizes that the same PlotTree can be reused for many questions, so the expensive hierarchical construction is amortized across multiple QA queries.

At the representation level, PlotTree is multimodal but text-centered. It preserves three grounded signals at the leaf level: recognized character identities, aligned dialogue, and a local plot summary. Higher levels store condensed plot summaries rather than raw visual features. This design reflects the paper’s claim that DVU requires maintaining identity bindings while also reasoning over abstract narrative structure.

3. Leaf-node construction and hierarchical condensation

PlotTree begins by sampling a video into $F$ keyframes. For each frame, InsightFace matches detected faces against the character library, yielding recognized characters $\mathcal{P}_i^0$. Those identities are rendered as colored bounding boxes, and the frame is paired with its aligned dialogue $d_i^0$ and a character-to-color map. LLaVA-1.6 then generates a grounded local plot summary $s_i^0$. Each leaf node is defined as

$n_i^0=(\mathcal{P}_i^0, d_i^0, s_i^0, t_i^0),$

where $t_i^0$ is the sequential keyframe index (Wu et al., 4 Jun 2026).

The flat leaf sequence $\mathcal{N}^0=\{n_i^0\}$ is then converted into a hierarchy by iterative bottom-up clustering. At level $l$, the node set $q$0 is partitioned into clusters $q$1 with a distance that balances temporal proximity and semantic similarity:

$q$2

Here, the temporal weight decays by level,

$q$3

with $q$4 and $q$5. The effect is that lower levels remain more temporally constrained, while higher levels become more semantically driven (Wu et al., 4 Jun 2026).

The node embeddings are defined differently at the leaf and non-leaf levels:

$q$6

Thus, the leaf embeddings encode explicit character identity, dialogue, and local summary, while upper levels encode only the condensed summary text. This separation is central to the system’s organization: fine-grained grounding is preserved at the bottom, and abstract plot semantics dominate higher up.

Each cluster $q$7 is sorted by time and summarized by Gemini-2.0-Flash into a parent summary $q$8. The parent node is

$q$9

with

$F$0

The number of clusters at each level is controlled by the compression rule

$F$1

and the experiments use $F$2 (Wu et al., 4 Jun 2026).

This construction yields a hierarchy in which leaf nodes retain local evidence and upper nodes capture larger plot arcs. The paper characterizes retrieval over this hierarchy as a way to capture “both macro-level themes and micro-level details simultaneously.”

4. Retrieval and question answering over the hierarchy

Once a PlotTree has been built, question answering proceeds by embedding the question $F$3 with the Qwen3 embedding model and comparing that embedding against the node embeddings across all tree levels. The system selects the top-$F$4 matching nodes; the implementation uses $F$5. The retrieved set is denoted $F$6 (Wu et al., 4 Jun 2026).

The corresponding node summaries are extracted, sorted chronologically, and concatenated into a single reasoning context. When timestamps are identical, higher-level nodes are given priority. That context is then combined with the multiple-choice QA prompt and sent to Gemini-2.0-Flash, or Gemini-3-Flash in the stronger-agent setting, to produce the final answer. The system therefore functions as a hierarchical retrieval-augmented generation pipeline over plot summaries rather than as an end-to-end trained VideoQA model.

A notable property of PlotTree is that it is essentially training-free as presented. The paper does not define a dedicated training objective, task-specific loss, or end-to-end supervised optimization procedure for PlotTree itself. Instead, it composes pretrained modules for face recognition, captioning, embedding, summarization, and answer generation. In practical terms, there is an offline stage—frame sampling, face recognition, subtitle alignment, captioning, clustering, and hierarchical condensation—and an online stage of question embedding, node retrieval, and answer reasoning.

5. Empirical results and ablation structure

PlotTree is evaluated on StoryVideoQA-G, a manually verified gold subset of 6,927 QA pairs, because API-based agent evaluation over the full 363K benchmark is costly. With Gemini-2.0-Flash, PlotTree achieves 86.50% accuracy on StoryVideoQA-G, compared with 83.63% for Video2RAG and 72.02% for VideoTree. It also exceeds the reported open-source baselines, including VideoLLaMA3 at 80.09%, Qwen3-VL at 78.48%, and ViLAMP at 77.34%. In the stronger-agent configuration using Gemini-3-Flash, PlotTree reaches 88.80%, slightly above standalone Gemini-3-Flash at 87.96% (Wu et al., 4 Jun 2026).

Setting	Accuracy
PlotTree + Gemini-2.0-Flash	86.50%
Video2RAG + Gemini-2.0-Flash	83.63%
VideoTree + Gemini-2.0-Flash	72.02%
PlotTree + Gemini-3-Flash	88.80%
Standalone Gemini-3-Flash	87.96%

The topic-level results indicate that PlotTree remains strong across both perception and inference categories. Reported accuracies include 83.07 on P-C, 85.58 on P-CL, 87.30 on P-CAL, 91.67 on I-C, 92.64 on I-A, 93.25 on I-AL, and 93.26 on I-CAL. The pattern suggests that the hierarchical representation is especially effective on inference-heavy questions, which is consistent with the original motivation of supporting storyline reasoning.

The ablation study isolates three modules: Ins. for face recognition and character identification, Dia. for dialogue, and Tree for the PlotTree hierarchy itself. Full PlotTree yields 86.50. The Tree-only configuration yields 82.37, Ins.-only 83.57, Dia.-only 83.79, and Ins.+Dia. without the Tree 83.63, matching Video2RAG under the paper’s standardized setup. Tree+Ins. yields 86.00 and Tree+Dia. yields 86.03. The authors interpret this as evidence that the Tree module is the principal structural gain, with identity and dialogue acting as complementary grounded evidence.

Character identification provides substantial gains across all agent baselines. Enabling it changes Video2RAG from 80.22 to 83.63, VideoTree from 72.02 to 75.99, and PlotTree from 82.37 to 86.50. The appendix also reports manual re-verification of 109,449 detected facial frames, with only 0.56% face-recognition error, and notes that correcting those errors changes PlotTree performance by only 0.04%. This is presented as evidence that the hierarchy suppresses some upstream noise rather than amplifying it (Wu et al., 4 Jun 2026).

6. Limitations, interpretation, and significance

PlotTree is explicitly framed as an answer to the inability of prior methods to maintain long-range character associations and coherent storyline understanding, but it does not eliminate all DVU bottlenecks. The paper identifies a persistent weakness on location-sensitive perception questions. It also acknowledges an inherent bottleneck in agent methods that rely on transformed captions or documents rather than direct raw-frame reasoning: summarization and captioning can discard information even when the subsequent reasoning architecture is strong (Wu et al., 4 Jun 2026).

The system also depends on multiple upstream components—face recognition, subtitle alignment, caption generation, semantic embedding, and LLM-based summarization—so its overall quality is conditioned on the reliability of those components. Moreover, its main agent evaluation is restricted to the StoryVideoQA-G subset because of API cost, and the paper does not provide an end-to-end learned optimization procedure that could jointly adapt all stages.

Within those limits, PlotTree’s significance lies in its representation choice. It treats long-form video not as an unordered memory bank of local clips but as a hierarchical plot object with grounded leaves and abstract internal summaries. The article’s central result is that this structural alignment with narrative form can outperform flatter retrieval systems and, in the stronger-agent setting, even slightly exceed direct use of a stronger multimodal model. PlotTree therefore occupies a specific position in long-video QA research: it is a modular, hierarchy-first agent for story understanding, built around the claim that long-range narrative reasoning is best served by a plot tree rather than by a flat context window.