MoRef-Attention for Efficient Long-Video Reasoning

• The paper introduces MoRef-attention, a parallel multi‐reference reasoning operator that splits long vision-token sequences into multiple short reference chunks for stable and efficient processing.
• MoRef-attention employs query-aware fusion to aggregate question-token activations across chunks, preserving detailed video information without the quadratic cost of full-sequence attention.
• Empirical results demonstrate that MoRef-attention significantly enhances performance and compute efficiency in Video-MLLMs, achieving longer-context understanding with lower overhead.

MoRef-attention is the central attention mechanism in Free-MoRef, a training-free inference strategy for Video Multimodal LLMs that enables reasoning over substantially longer videos than the native context window of the underlying LLM would ordinarily permit. In Free-MoRef, a long vision-token sequence is reconstructed into several short reference sequences, processed in parallel within one inference pass, and summarized through a query-aware fusion of question-token activations. The mechanism is designed to preserve as much video information as possible while avoiding the context-length failure and quadratic cost that arise when the entire long video is fed into the decoder at once (Wang et al., 4 Aug 2025).

1. Motivation and problem setting

MoRef-attention was introduced to address the context-length limitation of the underlying LLM in a Video-MLLM. Long videos generate too many vision tokens, and when the full sequence exceeds the LLM’s stable input length, performance degrades or memory is exhausted. The motivating contrast in Free-MoRef is with two established remedies. Token compression reduces sequence length but sacrifices detail and can lose important information. Streaming inference reuses KV cache over multiple passes, but inference latency grows with context length, so longer context implies slower reasoning (Wang et al., 4 Aug 2025).

Free-MoRef is formulated to obtain longer-context understanding in a single inference pass with low overhead. Its core idea is to split a long vision-token sequence into multiple short “references,” process them in parallel, and merge the results afterward. MoRef-attention is the operator that makes this parallel multi-reference reasoning possible. The paper explicitly describes the design as inspired by MoE-like multiplexing: rather than one long sequence, the model handles several short reference chunks, each acting like a different “expert view” of the same video (Wang et al., 4 Aug 2025).

This positioning is useful for avoiding two common misconceptions. First, MoRef-attention is not a compression method at the input stage; it reorganizes the input into multiple shorter full-information references rather than collapsing the entire video early. Second, it is not streaming inference; reasoning is multiplexed within one inference pass rather than accumulated across repeated passes (Wang et al., 4 Aug 2025).

2. Multi-reference reconstruction of vision tokens

Free-MoRef begins with multi-reference partitioning of the vision-token sequence. The long vision-token sequence is divided into $M$ temporal units, and within each temporal unit the sequence is further divided into $N$ fragments. Fragments from different units are then aggregated to produce $N$ reference chunks. Each reference chunk is therefore a short sequence containing tokens sampled across the whole video timeline, so the chunks collectively cover the full input while each individual chunk remains short enough for stable inference (Wang et al., 4 Aug 2025).

The paper notes several consequences of this construction. Larger $M$ means more temporal overlap between references. When $M=1$, the $N$ chunks are temporally independent. In the experimental setup, the method often uses $M=64$, and sets $N=2,4,8$ for $2\times$, $4\times$, and $N$0 context expansion, respectively (Wang et al., 4 Aug 2025).

After partitioning, the same system prompt and question are appended to each reference chunk. Each parallel decoder input therefore contains system prompt tokens, vision tokens for that reference, and question tokens. This arrangement is central to the later aggregation rule: the question is identical across references, whereas the visual evidence differs by chunk (Wang et al., 4 Aug 2025).

A plausible implication is that the partitioning scheme turns a single long temporal reasoning problem into several parallel partial-context reasoning problems, while preserving a common textual query that can serve as the locus of cross-reference aggregation. That implication follows directly from the fact that the system prompt and question are repeated across all references, whereas the vision tokens differ by reference.

3. Operator definition and layerwise computation

MoRef-attention replaces the standard self-attention layer in the LLM during these parallel runs. Its stated role is threefold: to attend within each reference chunk, to extract a response for the question from each chunk, and to aggregate the question-token activations across all references in parallel into a unified representation. The paper describes it as concurrently querying distinct references in parallel using an identical question and combining multiple attention outcomes to summarize a unified response for updating the question tokens within each decoder layer (Wang et al., 4 Aug 2025).

For the parallel chunks, the operator forms

$N$1

where $N$2 is the number of chunks or references, $N$3 is the sequence length, and $N$4 is the embedding dimension. Flash-attention is then applied, producing

$N$5

Because causal attention is used, $N$6 is identical across chunks, while $N$7 and $N$8 differ across chunks because each chunk contains different vision content. This yields the operator’s defining asymmetry: vision-token activations remain chunk-specific, whereas question-token activations are fused across chunks (Wang et al., 4 Aug 2025).

The exact fusion rule is

$N$9

Here $N$0 denotes the question-token output for reference chunk $N$1, and $N$2 is the weight assigned to that chunk. The weighted sum produces a unified question representation, and $N$3 broadcasts the same fused query response back to all chunks. The final MoRef-attention output is

$N$4

This construction makes MoRef-attention more specific than a generic “attention on multiple chunks.” It is a parallel attention-and-fusion operator in which each chunk is processed independently, chunk-specific visual evidence is preserved, and the question response is constrained to become a shared summary over all chunks (Wang et al., 4 Aug 2025).

4. Query-aware weighting and relation to other attention formulations

The chunk weights $N$5 are designed to be query-aware, because different references may matter differently depending on the question. Since FlashAttention does not expose attention weights, the authors estimate query-reference relevance manually using a separate cross-modal attention map: $N$6 The chunk weights are then computed as

$N$7

In the paper’s interpretation, $N$8 is the attention map for chunk $N$9, and the maximum value in that map is used as a proxy for how strongly the question attends to that chunk’s visual tokens. The weights are normalized across chunks. This constitutes the training-free gating scheme: no learned router is introduced; relevance is derived from attention itself (Wang et al., 4 Aug 2025).

MoRef-attention differs from standard self-attention in that standard self-attention processes the entire sequence jointly, allowing all tokens to attend to all earlier tokens, which becomes too expensive and may exceed context length in long-video settings. MoRef-attention instead splits the long sequence into $M$0 chunks, runs attention on each chunk in parallel, and fuses only the question activations across chunks. It therefore approximates full-sequence understanding without incurring full-sequence attention cost (Wang et al., 4 Aug 2025).

It also differs from classical cross-attention. Cross-attention ordinarily uses separate query and key/value sources, such as text querying vision features. MoRef-attention is not introduced as a separate cross-attention module inserted between modalities; it is built from the same self-attention machinery but applied to multiple reference sequences with a query-aware fusion over question outputs (Wang et al., 4 Aug 2025).

A useful comparative context is provided by Mixture-of-Head attention, or MoH, which interprets attention heads as experts and replaces the standard equal-weight head aggregation with a token-dependent sparse routing mechanism (Jin et al., 2024). MoRef-attention and MoH share a mixture-style perspective, but they operate at different granularities. MoH routes over heads for each token, whereas MoRef-attention multiplexes over reference chunks for a fixed question across partitioned video contexts. This suggests that MoRef-attention is structurally closer to context-level multiplexing than to head-level conditional computation.

5. Information loss, disrupted connectivity, and reference fusion

MoRef-attention has an explicit limitation. Because each reference chunk is processed separately, it breaks direct interactions between vision tokens that reside in different chunks. The paper states that the partition disrupts the connectivity among vision-tokens across different chunks. As a consequence, MoRef-attention alone is not fully equivalent to full attention, especially for dependencies spanning chunks, tasks requiring global temporal continuity, and questions whose answers require jointly combining information from multiple references (Wang et al., 4 Aug 2025).

Free-MoRef addresses this limitation through a later Reference Fusion step. The design draws on an observation from FastV: shallow decoder layers have more uniformly distributed vision attention, while deeper layers become more concentrated on the question token. On that basis, Free-MoRef keeps multi-reference reasoning in the shallow layers and then merges references at a chosen layer $M$1 (Wang et al., 4 Aug 2025).

The fusion procedure is defined as follows. First, it computes the attention map $M$2 from the query tokens to the vision tokens. Second, it averages $M$3 over the question-token dimension to obtain an importance matrix

$M$4

where each entry estimates the importance of a vision token. Third, it prunes $M$5 of the least important vision tokens within each chunk. Fourth, it aggregates the remaining tokens from all chunks into a global reference according to temporal order. Fifth, it carries over system prompt and question tokens directly. After that point, the default decoder layers operate only on the global reference (Wang et al., 4 Aug 2025).

This establishes a two-stage computation pattern. MoRef-attention performs parallel chunk reasoning while the model still benefits from multiplexed context coverage; Reference Fusion then restores global connectivity and further reduces computation. A plausible implication is that the architecture intentionally separates coarse distributed evidence gathering from later consolidated reasoning, rather than trying to preserve all cross-reference interactions from the outset.

6. Empirical behavior and reported significance

The paper’s ablations identify MoRef-attention as the main contributor to the performance gain. In the component ablation reported in Table 4, the baseline without partition, MoRef-attention, or fusion scores 64.9; only reference fusion scores 63.9; partition plus fusion but no MoRef-attention scores 62.0; partition plus MoRef-attention but no fusion scores 65.8; and full Free-MoRef scores 66.3. The authors interpret this to show that partitioning alone is not helpful, reference fusion alone is not sufficient, MoRef-attention provides the major gain, and fusion adds a further boost on top (Wang et al., 4 Aug 2025).

The chunk-number ablation reports the following pattern: with $M$6, FLOPs are 100% and overall performance is 64.9; with $M$7, FLOPs are 27.6% and overall performance is 66.3; with $M$8, FLOPs are 25% and overall performance is 66.1; with $M$9, FLOPs are 23.6% and overall performance is 65.9. On these figures, MoRef-style multiplexing yields large compute savings with no loss and even an improvement at $M=1$0 (Wang et al., 4 Aug 2025).

The long-video results reported for Free-MoRef indicate full perception of $M=1$1 to $M=1$2 longer input frames without compression on a single A100 GPU while keeping instant responses. The paper reports improved performance on VideoMME, MLVU, and LongVideoBench, and notes that in some settings the method surpasses dedicatedly trained long-video-MLLMs (Wang et al., 4 Aug 2025).

These findings delimit the mechanism’s significance. MoRef-attention is not presented merely as a speed optimization; it is the component that makes single-pass multiplexed long-context perception effective. The empirical pattern also clarifies an important point of interpretation: the gains do not arise simply from partitioning long videos into smaller pieces. They arise from the specific parallel attention-and-fusion rule that shares a unified query representation across chunk-specific visual evidence.

7. Conceptual position within efficient attention research

Within the broader attention literature, MoRef-attention occupies a distinct position. It is neither a head-routing mechanism such as MoH, which treats attention heads as experts with token-dependent Top-$M=1$3 selection (Jin et al., 2024), nor a multiresolution approximation of the attention matrix such as MRA attention, which uses a hardware-friendly overcomplete block basis and greedy coarse-to-fine selection to approximate self-attention (Zeng et al., 2022). Instead, MoRef-attention targets the context-length bottleneck in Video-MLLMs by restructuring the input into multiple short references and aggregating question responses across them (Wang et al., 4 Aug 2025).

Its central design principle is therefore not sparsification of matrix entries, low-rank factorization, or routing over heads. It is reference-level multiplexing. The paper’s own concise summary describes it as a training-free, parallel multi-reference attention mechanism that lets a Video-MLLM reason over long videos by splitting the vision-token sequence into multiple short reference chunks, running attention on all chunks in parallel, and fusing the question-token activations into a shared query representation (Wang et al., 4 Aug 2025).

This framing is also relevant for assessing its limitations. Because MoRef-attention initially neglects cross-reference vision interactions, it should not be treated as a strict replacement for full self-attention in all regimes. Its effectiveness depends on the later Reference Fusion stage and on the assumption that a shared question-token summary can capture enough global evidence during the shallow-layer multiplexed phase. The reported results indicate that this assumption is often practically effective for long-video understanding, while the paper’s own analysis keeps the tradeoff explicit rather than claiming full equivalence to unrestricted attention (Wang et al., 4 Aug 2025).