--- title: Multi-Video Structured Prompt url: https://www.emergentmind.com/topics/multi-video-structured-prompt type: topic --- # Multi-Video Structured Prompt Searching arXiv for the cited works to ground the article in current literature. Multi-Video Structured Prompt is a family of prompting and conditioning paradigms in which multiple temporally or semantically related video units are not treated as an undifferentiated token stream, but as a structured object with explicit internal organization. Across recent work, the term covers several distinct but converging settings: temporally ordered prompt scripts for long-form video generation, graph-structured prompting for multi-video reasoning, structured text supervision over adjacent action clips, prompt banks for multi-shot extrapolation, and video-conditioned prompting via reference clips. The common premise is that naive concatenation—whether of prompts, video tokens, or clip descriptions—fails to preserve coherence, semantic alignment, or computational efficiency, whereas structured decomposition into segments, roles, graphs, or prompt fields enables better continuity, reasoning, and control [2412.18597]. ## 1. Conceptual scope and formalization In video generation, Multi-Video Structured Prompt is often framed as a temporal composition problem rather than a simple multi-text conditioning problem. DiTCtrl formulates the setting as: given a pretrained single-prompt text-to-video diffusion model \(\mathcal{F}\) and an ordered prompt sequence \(\{P_1, P_2, \dots, P_n\}\), generate a coherent long video that follows these prompts over time. The paper writes this as \[ \mathcal{V}_{\{1,...,n\} = \mathrm{DiTCtrl}\{\mathcal{F}(P_1),...,\mathcal{F}(P_n)\}. \] Here, each prompt governs a semantic phase or segment, and the central requirement is not simultaneous blending but temporally ordered realization with continuity of subject identity, motion, and scene evolution across segment boundaries [2412.18597]. In video language modeling, the same general idea appears in a different form. The early version of the structured multi-video collaborative reasoning framework takes as input a target video \(V_0\) and related videos \(\{V_1, V_2, \dots, V_N\}\), and argues that direct concatenation of all raw video tokens is counterproductive. Instead, it converts each video into a spatio-temporal graph, fuses target and related graphs, and serializes the result as a structured prompt containing target-video visual tokens, target-video graph tokens, related-video graph tokens, the question, and textual guidance, with the fixed Chain-of-Thought cue “Think through the process step by step” [2509.13161]. Bridge-Prompt provides an earlier precursor in instructional video understanding. Although it operates within one long instructional video rather than across separate videos, it explicitly converts a local sequence of adjacent actions into a hierarchy of text prompts: statistical, ordinal, semantic, and integrated. The integrated prompt \[ y_{integ} = y^1_{sem} \oplus y^2_{sem} \oplus \cdots \oplus y^K_{sem} \] is a textual serialization of a local action sequence and shows that structured prompting can encode order, count, and composition rather than only clip-local labels [2203.14104]. Across these formulations, Multi-Video Structured Prompt denotes a shift from flat prompt strings or flat token concatenation toward prompt structures that preserve relations among video units: temporal order, semantic role, graph connectivity, narrative dependency, or prompt-source identity. This suggests a unifying interpretation: the “prompt” is no longer merely text, but an organized interface between multiple video-conditioned evidence sources and a downstream generator or reasoner. ## 2. Temporal composition for long-form video generation DiTCtrl is one of the clearest formulations of structured prompting for generation under MM-DiT architectures. It argues that multi-prompt video generation should be treated as temporal video editing with smooth transitions rather than naive prompt concatenation. The method is training-free: no finetuning, retraining, or optimization is performed for the multi-prompt task; all control is injected at inference time by modifying the pretrained MM-DiT attention behavior and latent composition [2412.18597]. The architectural observation behind DiTCtrl is that MM-DiT full attention over concatenated text and video tokens can be decomposed into four interpretable regions: video-to-video, text-to-text, text-to-video, and video-to-text. The text-to-video and video-to-text regions behave similarly to cross-attention maps in UNet-based diffusion models, while video-to-video behaves like spatiotemporal self-attention. This makes training-free attention control possible in a Sora-like text-to-video model, concretely implemented on CogVideoX-2B [2412.18597]. Its core control mechanism is mask-guided key/value sharing across adjacent prompt segments. At denoising step \(t\), DiTCtrl extracts cross-modal regional attention maps, \[ {A^c_i} \in \mathbb{R}^{F\times H\times W \times N}, \] derives prompt-specific masks \(M_{i-1}\) and \(M_i\), and then uses current-segment queries to attend to previous-segment keys and values separately for object and background regions: \[ f^l_{o} = \text{Attention}(Q^l_{i}, K^l_{i-1}, V^l_{i-1}; M_{i-1}), \quad f^l_{b} = \text{Attention}(Q^l_{i}, K^l_{i-1}, V^l_{i-1}; 1 - M_{i-1}), \] \[ \bar{f}^l = f^l_{o} * M_{i} + f^l_{b} * (1 - M_{i}). \] This enforces object-to-object and background-to-background transfer, preserving appearance consistency while still allowing semantic change [2412.18597]. To smooth boundaries visually, DiTCtrl uses overlapped latent windows between adjacent segments. If \(\mathcal{V}_{i-1}\) and \(\mathcal{V}_i\) are adjacent generated clips and \(T\) frames overlap, the latent at overlap position \(t\) is blended as \[ \mathbf{z}_t = w(t) \cdot \mathbf{z}_{t,i-1} + (1 - w(t)) \cdot \mathbf{z}_{t,i}, \] with symmetric triangular weight \[ w(t) = \min\left(\frac{2(t + 0.5)}{T}, 2 - \frac{2(t + 0.5)}{T}\right), \quad t \in [0, T-1]. \] The overlap thus becomes an explicit transition window between prompt-defined phases [2412.18597]. The experimental evidence is centered on MPVBench, a benchmark containing 130 long-form prompt sequences spanning 10 transition modes. The paper introduces CSCV, the Clip Similarity Coefficient of Variation, defined using adjacent-frame similarities \[ s_i = \mathbf{x}_i^\top \mathbf{x}_{i+1}, \quad i = 1,\ldots,n-1 \] and score \[ \text{score} = \frac{1}{1 + \lambda \cdot \frac{\sigma(s)}{\mu(s)}}. \] On MPVBench, DiTCtrl achieves the best CSCV at 84.90%, compared with 84.37% for FreeNoise, 78.74% for FreeNoise+DiT, 74.97% for Video-Infinity, and 59.44% for Gen-L-Video. It also achieves the best motion smoothness score at 97.80%, though its text-image similarity is 30.68%, lower than FreeNoise at 32.69% and Video-Infinity at 32.35%, which the paper interprets as a continuity–prompt-adherence tradeoff caused by KV-sharing [2412.18597]. The paper’s broader implication is explicit: structured prompt sequences should be temporally ordered and locally compatible. Adjacent prompts should differ in controlled ways—action, style, camera behavior, or location—while preserving a specific consistent object or visual thread when possible. This makes “multi-video structured prompt” synonymous, in this line of work, with inference-time temporal composition over prompt-defined video segments [2412.18597]. ## 3. Structured prompts for multi-video reasoning and retrieval The multi-video collaborative reasoning framework for Video LLMs addresses a different failure mode: spatio-temporal incompleteness in individual videos. Its central claim is that simply feeding more raw videos into an LLM is harmful. On InternVid-QA, the “multi-video tokens” baseline inflates context length from 2.1K to 12.5K tokens and drops accuracy from 45.6 to 28.5. By contrast, the proposed graph-fusion prompt reaches 49.5 accuracy with only 2.3K tokens [2509.13161]. The foundation of this framework is the Video Structuring Module. Each video is converted into a spatio-temporal graph by scene detection, dense captioning, scene-graph parsing, grounding, and tracking. Textual scene graphs are represented as triplets \[ \tau_i=\{s_i, p_i, o_i\}, \] or \(\langle \text{subject} - \text{predicate} - \text{object} \rangle\). Grounding produces \[ \mathcal{B}=\{(b_{s_i}, b_{o_i}) \mid 1\leq i\leq M\}, \] and tracking produces \[ \mathcal{T}=\{\mathrm{Tid}_{s_i}, \mathrm{Tid}_{o_i}\}. \] Each grounded object region is cropped and encoded with OpenCLIP, using only the \([\mathrm{CLS}]\) token as the node feature [2509.13161]. The Graph Fusion Module then performs two operations. First, a Hierarchical Frame Graph Attention Network propagates intra-video spatio-temporal structure through graph edges. Second, Cross-Graph Attention lets target-video nodes absorb useful information from related videos. With target features \(\mathbf{X}_{tar}\) and related features \(\mathbf{X}_{rel}\), the target query is \[ \mathbf{Q}_{tar}=\mathbf{X}_{tar}\mathbf{W}_Q, \] while keys and values come from concatenated target and related features: \[ [\mathbf{V}_{tar}, \mathbf{V}_{rel}] = [\mathbf{X}_{tar}, \mathbf{X}_{rel}] \mathbf{W}_V, \] \[ [\mathbf{K}_{tar}, \mathbf{K}_{rel}] = [\mathbf{X}_{tar}, \mathbf{X}_{rel}] \mathbf{W}_K. \] A learnable class embedding distinguishes target and related nodes: \[ \mathrm{CE}_{tar} = \sigma(\boldsymbol{\alpha}), \qquad \mathrm{CE}_{rel} = 1 - \sigma(\boldsymbol{\alpha}). \] These class embeddings are added to attention inputs before fusion [2509.13161]. The structured prompt sent to the LLM interleaves target-video visual tokens, target-video graph tokens, related-video graph tokens, task instructions, the user question, and the fixed CoT cue. Crucially, only the target video retains raw visual tokens; related videos are represented only by graph tokens. This asymmetry is the central prompt-efficiency strategy [2509.13161]. Empirically, the framework is implemented on top of Video-LLaVA with Vicuna-7B v1.5 and achieves 61.8 accuracy on MSRVTT-QA, 46.9 on ActivityNet-QA, and 49.5 on InternVid-QA, improving over Video-LLaVA by \(+2.6\%\), \(+1.6\%\), and \(+3.9\%\), respectively. Ablations show that removing graph structure or cross-graph fusion harms performance: direct projection of graph features yields 46.2 on InternVid-QA, adding HF-GAT lifts it to 47.8, and adding CGA yields 49.5, while adding an FFN afterward reduces accuracy to 48.7 [2509.13161]. A related but retrieval-oriented formulation appears in MAVIS, which rethinks text-to-video retrieval as structured query execution over a corpus-level Structured Semantic Library. Each video is parsed into a concise caption and a tuple \[ \mathbf{s}_i = \big( s_i^{scn}, s_i^{obj}, s_i^{act} \big), \] with sub-libraries indexed as \[ \mathcal{L}_{d}[k] = \big\{ (\text{id}=i, \text{cap}=c_i) \mid k \in s_i^{d} \big\}, \quad d \in \{scn, obj, act\}. \] A planner decomposes a query into active semantic dimensions, \[ \mathcal{Q} = \{q^r \mid r \in \mathcal{R}\}, \qquad \mathcal{R} \subseteq \{scn, obj, act\}, \] specialized agents retrieve candidates, and a Logic-aware Debate applies a strict veto protocol before final visual verification [2606.09641]. The debate mechanism is formalized with agent proposal sets \[ M_r = \big\{ (v_i, \phi_r(v_i)) \mid \phi_r(v_i) > \tau_{\text{soft}} \big\}, \] pool \[ \mathcal{V}_{\text{pool}} = \bigcup_{r \in \mathcal{R}} \{v_i \mid (v_i, \phi_r) \in M_r\}, \] and controversial set \[ \mathcal{V}_{\text{contro}} = \mathcal{V}_{\text{pool}} \setminus \big\{ v_i \in \mathcal{V}_{\text{pool}} \mid \exists r, \phi_r(v_i) < \tau_{\text{hard}} \big\}. \] On MSR-VTT, MSVD, and ActivityNet, MAVIS achieves 78.6/94.9/96.3, 78.1/94.25/97.33, and 69.15/92.4/96.8 for R@1/R@5/R@10, respectively, without task-specific finetuning [2606.09641]. Together, these works establish a general principle: when multiple videos are involved, structured prompts outperform raw multi-video token concatenation because they compress evidence into semantically typed units before LLM reasoning. ## 4. Sequence- and task-structured prompting in video representation learning Bridge-Prompt shows that structured prompting can also serve as supervision rather than merely inference-time control. It reformulates adjacent action labels in instructional videos into a “three-plus-one-level” prompt engineering scheme. The statistical prompt is > “this video clip contains \underline{\{num(K)\} actions in total” and is denoted \(y_{stat}\). The ordinal prompt is > “this is the \underline{\{ord_i\} action in the video” and the semantic prompt is > “\underline{\{ord_i\}, the person is performing the action step of \underline{\{vp_i\}” with integrated prompt given by \[ y_{integ} = y^1_{sem} \oplus y^2_{sem} \oplus \cdots \oplus y^K_{sem}. \] The model then aligns these prompt levels with video cuts containing \(K\) consecutive actions through contrastive losses [2203.14104]. Its feature extraction is based on a frame-wise encoder \(\mathcal{F}_I\), a text encoder \(\mathcal{F}_T\), and a fusion encoder \(\mathcal{F}_F\). For the \(i\)-th action, the clip representation is \[ z^i_c = \mathcal{F}_F(\mathcal{F}_I(f_1), \ldots, \mathcal{F}_I(f_{L_c}), z^i_{ord}), \] and similarity is computed by cosine similarity \[ s(z_x, z_y) = \frac{z_x \cdot z_y}{|z_x|\,|z_y|}. \] The total loss combines semantic, integrated, and statistical alignment terms [2203.14104]. Empirically, Bridge-Prompt improves action segmentation and long-term activity recognition. On GTEA with ASFormer, it achieves F1@10 94.1, F1@25 92.0, F1@50 83.0, Edit 91.6, and Acc 81.2. On 50Salads, it achieves F1@10 89.2, F1@25 87.8, F1@50 81.3, Edit 83.8, and Acc 88.1. On Breakfast, Bridge-Prompt reaches 80.00% accuracy for long-term activity recognition [2203.14104]. The loss ablation shows that adding integrated and statistical prompt supervision materially improves performance beyond semantic prompts alone [2203.14104]. A different form of structured prompting appears in PromptonomyViT, where prompts encode task structure rather than natural-language sequence structure. A fixed set of learned task prompt vectors \[ p_1,p_2,\cdots,p_n\in \mathbb{R}^{1\times d} \] is appended to the transformer token sequence: \[ z = \left[z_{CLS},z_1,z_2,...,z_N,p_1,p_2,...,p_n\right]. \] Each prompt corresponds to a synthetic scene-level task such as depth, normals, semantic segmentation, 3D pose, or boxes, and these prompt streams interact with video patch tokens through the entire transformer [2212.04821]. Predictions are made from the downstream CLS token, \[ \hat{Y}_{CLS} = H_{CLS}(F_{CLS}(z)), \] and from task-specific prompt tokens, \[ \hat{Y}_{i} = H_i(F_{P_i}(z), F_{PT}(z)). \] The training objective combines downstream classification with synthetic task losses: \[ \mathcal{L}_{Total} = \lambda_{DT}\mathcal{L}_{DT} + \lambda_{Depth}\mathcal{L}_{Depth} + \lambda_{Normal}\mathcal{L}_{Normal} +\lambda_{Segm}\mathcal{L}_{Segm} + \lambda_{3DPose}\mathcal{L}_{Pose3d} + \lambda_{Boxes}\mathcal{L}_{Boxes}. \] On SomethingElse compositional recognition, PViT reaches 65.5 top-1 / 89.0 top-5 versus 63.3 / 87.5 for MViTv2; on Ego4D object state change classification, it improves top-1 from 71.6 to 74.8; on Diving48 it improves from 73.1 to 85.8; and on AVA it improves mAP from 26.8 to 28.4 [2212.04821]. DLM-VMTL extends the idea to heterogeneous video multi-task prompt learning. For each auxiliary task, cross-task prompt extraction is performed by \[ [P_{Aux}^i,E_{middle+1}]=L([P,E_{middle}]), \quad i = 1,2,...,N, \] followed by a dimension corresponding unit \[ P_{Aux}^i = FC_2(FC_1(P_{Aux}^i)), \] adapter-based alignment \[ P_{Pri}^i = Adapter(P_{Aux}^i), \] concatenation \[ P_{Pri}=concat(P_{Pri}^1,P_{Pri}^2,...,P_{Pri}^N), \] and injection into the primary task model: \[ [P,E_{middle+1}^{Pri}]=L_{Pri}([P_{Pri},E_{middle}^{Pri}]). \] Across six tasks, DLM-VMTL surpasses full finetuning while using 10.8% of total parameters: for example, on SSv2 it reaches 76.7 versus 75.2 for full finetuning, on AVA 42.2 versus 40.4, and on YouTube-VIS 2019 66.1 versus 64.9 [2408.16195]. These works collectively broaden Multi-Video Structured Prompt beyond prompt strings. In representation learning, the prompt can be a structured sequence supervision signal, a bank of task tokens, or a task-organized transfer interface. ## 5. Prompt banks, recursive context, and multi-shot narrative generation Later work makes the structure increasingly explicit at the shot level. PACR-Video formulates multi-shot long video extrapolation as sequential generation of ordered shots \[ \mathcal{V}=\{v_1,\ldots,v_T\} \] conditioned on shot-level prompts \[ \mathcal{Y}=\{y_1,\ldots,y_T\}. \] At step \(t\), the model conditions on \[ c_t=[e(y_t);q_t;p_t^{\mathrm{role}}], \] where \(e(y_t)\) is the current shot text encoding, \(q_t\) is routed historical prompt context, and \(p_t^{\mathrm{role}}\) is a learned shot-role embedding [2607.06481]. The persistent memory is a recursive prompt bank: \[ \mathcal{B}_t=\{b_i^{e},b_i^{l},b_i^{a},b_i^{s}\}_{i\le t}, \] storing compact entity, location, action, and style prompts for each prior shot. Narrative dependency routing is defined by \[ \rho_{t,i,k} = \operatorname{softmax}_{i,k} \left( w_k^\top \phi\!\left[ p_t^{\mathrm{role}}, e(y_t), b_i^k, \Delta(t,i) \right] \right), \qquad q_t = \sum_{i