---
title: 'Free-MoRef: Training-Free Long-Video Inference'
url: https://www.emergentmind.com/topics/free-moref
type: topic
---

# Free-MoRef: Training-Free Long-Video Inference

Searching arXiv for the Free-MoRef paper and closely related long-video Video-MLLM context.
Free-MoRef is a training-free inference method for long-video understanding in Video Multimodal Large Language Models (Video-MLLMs). It is designed to extend effective context perception within a single forward pass by reconstructing an overlong visual token sequence into several shorter “references,” processing those references in parallel with a modified attention mechanism called MoRef-attention, and then fusing the references at an intermediate decoder layer so that some cross-reference interactions can be recovered before generation continues [2508.02134]. In the formulation reported for LLaVA-Video-7B, Free-MoRef is presented as an alternative to token compression and streaming inference, with the stated aim of achieving fuller perception of more frames without compression while retaining the feel of instant response because everything is handled in one inference [2508.02134].

## 1. Problem setting and conceptual role

Free-MoRef is motivated by a central systems bottleneck in Video-MLLMs: videos generate substantially more tokens than images, while the underlying decoder-only LLM remains constrained by a practical context limit and by the quadratic cost of self-attention [2508.02134]. The reported example is LLaVA-Video with Qwen2 as the LLM, where a threshold around \(32{,}768\) tokens is described as the point for stable inference; the default setup uses up to \(64\) frames, each represented by \(182\) tokens, or about \(11{,}648\) visual tokens before adding text, whereas \(256\) or \(512\) frames would push the token count toward about \(46\text{k}\) or \(93\text{k}+\) [2508.02134].

Within that setting, Free-MoRef is positioned against two dominant solution families. Token compression reduces the number of visual tokens by pruning or summarization, but the trade-off is loss of feature granularity and information fidelity. Streaming inference preserves detail by processing the video in segments while reusing historical information such as KV-cache, but latency scales with context length because the model must process the video in multiple stages [2508.02134]. Free-MoRef is framed as a third option: instead of compressing a long video into fewer tokens or serializing chunk processing across time, it creates multiple parallel short-context views of the long video and allows the same question token to query all of them at once [2508.02134].

The method is described as “training-free,” meaning that it requires no retraining of the base Video-MLLM, no finetuning to support longer contexts, and no learned routing or fusion parameters specific to the method. What changes is the organization of inference: the visual tokens are repartitioned, attention is modified across references, and hidden activations are fused at an intermediate stage [2508.02134]. This makes the method immediately deployable on an existing Video-MLLM in the sense claimed by the paper.

The name “Free-MoRef” is explicitly interpreted as a training-free Mixture-of-References inference method. The paper states that it is inspired by Mixture-of-Experts, but the analogy is limited: there are no separately trained expert subnetworks, all references share the same model weights, and what differs across branches is the context slice each branch sees rather than the parameters themselves [2508.02134]. This suggests that the central novelty lies in input-side multiplexing of context perception rather than parameter-side specialization.

## 2. Multi-reference partition and inference organization

The first stage of Free-MoRef is Multi-Reference Partition. An overlong visual token sequence is reconstructed into \(N\) shorter sequences called references or chunks [2508.02134]. Each reference contains the same system prompt, a subset or reconstruction of visual tokens, and the same question. The references are intended to be partial but informative abstractions of the full long video [2508.02134].

The partition is controlled by two hyperparameters. The whole visual token sequence is first divided temporally into \(M\) units, and each unit is then decomposed into \(N\) fragments. Fragments from different units are aggregated to form \(N\) reference chunks [2508.02134]. The procedure is described in the data as follows:

\[
R_r = \mathrm{Concat}\big(f_{1,r}, f_{2,r}, \dots, f_{M,r}\big), \qquad r=1,\dots,N
\]

This reconstructed expression is explicitly identified in the source material as a faithful reconstruction from the text rather than a printed equation. Its significance is architectural: each reference can contain tokens drawn from across the full video timeline rather than from one contiguous segment, depending on \(M\) [2508.02134]. The paper interprets \(M=1\) as temporally independent large chunks, whereas larger \(M\) creates stronger temporal interleaving across references [2508.02134].

Each reference sequence is written as

\[
X_r = [X^{sys}; X^{vis}_r; X^{ques}],
\]

and stacking all references yields

\[
\boldsymbol{X}\in \mathbb{R}^{N\times l\times D},
\]

where \(N\) is the number of references, \(l\) is the per-reference sequence length, and \(D\) is the embedding dimension [2508.02134]. All chunks share the same system and question tokens; only the visual tokens differ.

The operational claim made for this organization is that the same base model that can reason over one short sequence is repurposed to reason over several short sequences in parallel, after which their evidence is merged [2508.02134]. This is the sense in which Free-MoRef is said to “instantly multiplex” context perception capabilities.

## 3. MoRef-attention and reference fusion

MoRef-attention is the core mechanism of the method. The paper states that the LLM’s self-attention is modified so that multiple references are processed in parallel, while the question-token updates are fused across references [2508.02134]. The tensors are given as

\[
\boldsymbol{Q}, \boldsymbol{K}, \boldsymbol{V} \in \mathbb{R}^{N \times l \times D},
\]

and standard attention is run within each chunk, producing outputs decomposed as

\[
\boldsymbol{O} = [\boldsymbol{O}^{sys}, \boldsymbol{O}^{vis}, \boldsymbol{O}^{ques}].
\]

Because the system prompt is identical in all references, \(\boldsymbol{O}^{sys}\) is reported as identical across chunks, whereas \(\boldsymbol{O}^{vis}\) and \(\boldsymbol{O}^{ques}\) differ because the visual tokens differ [2508.02134].

The defining operation is fusion of question outputs across references:

\[
\boldsymbol{O}^{fusion} = \Big(\sum_{i=1}^{N}\omega_i\cdot\boldsymbol{O}^{ques}_{i}\Big).\mathrm{repeat}(N), \qquad \sum_{i=1}^{N}\omega_i=1.
\]

The fused query activation is copied back to every chunk, so subsequent layers begin from the same query representation in all branches [2508.02134]. The final output of MoRef-attention is then

\[
\boldsymbol{O}^{MoRef} = [\boldsymbol{O}^{sys}, \boldsymbol{O}^{vis}, \boldsymbol{O}^{fusion}].
\]

This means that the system tokens remain unchanged, the vision hidden states remain chunk-specific, and the question hidden states become globally mixed across references [2508.02134].

The weights \(\omega_i\) are derived from an auxiliary cross-modal attention map between question queries and vision keys:

\[
\boldsymbol{A} = \mathrm{softmax}(\boldsymbol{Q}^{ques}\times (\boldsymbol{K}^{vis})^T ).
\]

Because the implementation is stated to use FlashAttention, which does not directly expose attention weights, this auxiliary map is computed manually [2508.02134]. The reference weight for chunk \(i\) is then

\[
\omega_i = \frac{\max(\boldsymbol{A}[i])}{\sum_{i=1}^{N}\max(\boldsymbol{A}[i])}.
\]

Thus, each chunk contributes to the fused question activation in proportion to its strongest query-to-vision match [2508.02134].

The method also introduces a later Reference Fusion stage. The paper states that after the shadow layers in the LLM, attention becomes more concentrated on query tokens, and on that basis the model stops parallel multi-reference processing at an intermediate decoder layer \(L\), selects important visual tokens from each chunk, merges them into a single global sequence, and continues the remaining decoder layers with standard attention [2508.02134]. The term “shadow layers” is described in the source summary as likely referring operationally to shallow early decoder layers; this interpretation is explicitly marked as explanatory rather than as a verbatim statement from the paper.

For reference fusion, the same auxiliary attention map is used, now with shape

\[
\boldsymbol{A}\in\mathbb{R}^{N\times l_{ques}\times l_{vis}},
\]

and averaging over the question dimension yields a vision-token importance matrix

\[
\boldsymbol{E}\in\mathbb{R}^{N\times l_{vis}}.
\]

The summary reconstructs the implied formula as

\[
E_{ij} = \frac{1}{l_{ques}}\sum_{t=1}^{l_{ques}} A_{i,t,j},
\]

again explicitly as a faithful reconstruction from the text [2508.02134]. Based on \(\boldsymbol{E}\), the least important \(1-\frac{1}{N}\) fraction of visual tokens in each chunk is pruned, so that approximately a \(1/N\) fraction survives from each reference [2508.02134]. The retained tokens are then merged in temporal order into a global reference, summarized in the source as

\[
R^{global} = \mathrm{TemporalMerge}\big(\mathrm{TopK}(R_1,E_1), \dots, \mathrm{TopK}(R_N,E_N)\big).
\]

The reason given for this step is that MoRef-attention does not model direct interactions among visual tokens from different references in shallow layers, whereas reference fusion brings key visual tokens into one sequence so that ordinary self-attention in later layers can capture cross-reference dependencies [2508.02134].

## 4. Computational properties and scaling behavior

The paper argues that Free-MoRef reduces the effective attention burden by partitioning a long visual sequence into \(N\) non-overlapping chunks, which lowers attention complexity by approximately a factor of

\[
\frac{1}{N}
\]

relative to full attention over the whole sequence [2508.02134]. The stated intuition is the usual one: if full attention over sequence length \(L\) scales like \(L^2\), then \(N\) chunkwise attention computations over length \(L/N\) cost \(N\cdot (L/N)^2 = L^2/N\) [2508.02134].

This scaling argument is paired with concrete FLOP comparisons for LLaVA-Video-7B. Relative to the 64-frame baseline at 100% FLOPs, the paper reports the following values [2508.02134]:

| Setting | FLOPs |
|---|---:|
| 64 frames baseline | 100% |
| 128 frames naive | 400% |
| 128 frames + Free-MoRef | 110.4% |
| 256 frames naive | 1600% |
| 256 frames + Free-MoRef | 163.2% |
| 512 frames naive | 6400% |
| 512 frames + Free-MoRef | 400% |

An ablation for 128 frames with \(N=2\) reports 27.6% FLOPs under the chunked setting used in that experiment [2508.02134]. The paper interprets these results as evidence that naive long-context scaling is prohibitive, whereas chunked parallel references keep the growth much milder.

The system claim attached to these efficiency numbers is that Free-MoRef can achieve full perception of \(2\times\) to \(8\times\) longer input frames without compression on a single A100 GPU while keeping instant responses [2508.02134]. The conclusion also states that the method reached comprehensive understanding of 1024 frames on a 7B Video-LLM with one A100 [2508.02134]. The term “instant” is used comparatively, to distinguish this single-pass parallel reasoning regime from streaming inference, in which longer context implies additional serial stages and correspondingly larger first-token latency [2508.02134].

## 5. Experimental results and empirical profile

The reported empirical evaluation uses LLaVA-Video-7B on VideoMME, MLVU, and LongVideoBench, with lmms-eval on a single A100 GPU [2508.02134]. The default frame-loading setup is FPS \(=1\), up to 64 frames, and 182 tokens per frame. Extended settings use 128, 256, and 512 frames; the paper reports \(M=64\), \(N=2,4,8\) for the \(2\times\), \(4\times\), and \(8\times\) settings respectively, with reference fusion layers \(L=3,6,12\) [2508.02134].

For the main long-context study on LLaVA-Video-7B, the reported benchmark results are as follows [2508.02134]:

| Setting | MLVU | VideoMME overall | LongVideoBench overall |
|---|---:|---:|---:|
| 64 frames baseline | 70.3 | 64.3 | 58.8 |
| 128 frames naive | 70.2 | 64.9 | 58.7 |
| 128 frames + Free-MoRef | 70.8 | 66.3 | 59.3 |
| 256 frames naive | 67.2 | 63.1 | 56.7 |
| 256 frames + Free-MoRef | 72.5 | 66.3 | 59.3 |
| 512 frames naive | 61.1 | 60.6 | 54.3 |
| 512 frames + Free-MoRef | 72.8 | 66.9 | 59.9 |

These numbers are central to the paper’s claim. Naively increasing frames provides little gain at 128 frames and degraded performance at 256 and 512 frames, whereas Free-MoRef improves performance as context length increases because, on the paper’s interpretation, it can exploit the additional information without overwhelming the model [2508.02134].

The comparison table against other models around the 7B-8B scale reports the following headline numbers: InternVL2 at 64.0 / 54.6 / 54.0 on MLVU / LongVideoBench / VideoMME overall; InternVL2.5 at 68.4 / 57.5 / 64.5; Qwen2-VL at 64.8 / 55.6 / 63.3; LLaVA-OneVision at 64.7 / 56.3 / 58.2; LLaVA-Video at 70.2 / 58.2 / 64.3; Kangaroo at 61.0 / 54.8 / 56.0; LongVILA at – / 57.1 / 60.1; LongVA at 56.3 / – / 52.6; Video-XL at 64.9 / 50.7 / 55.5; RETAKE at 69.8 / – / 63.9; and LLaVA-Video with Free-MoRef at 72.8 / 59.9 / 66.9 [2508.02134]. On the VideoMME long subset, the paper notes that Free-MoRef reaches 56.0, slightly below RETAKE’s 56.2, while still giving the best overall model on VideoMME overall and the highest scores in the table on MLVU and LongVideoBench [2508.02134].

The ablation studies are also structurally important. The component ablation reports 64.9 for the 128-frame setting without partition, MoRef, or fusion; 63.9 for fusion only; 62.0 for partition with full-attention-separate and fusion but no MoRef; 65.8 for partition with MoRef and no fusion; and 66.3 for partition with MoRef and fusion [2508.02134]. This shows that partitioning alone does not account for the performance gain; the performance increase is attributed primarily to MoRef-attention, with reference fusion adding a further boost [2508.02134].

The study of the number of references \(N\) at 128 frames reports 64.9 at \(N=1\) with 100% FLOPs, 66.3 at \(N=2\) with 27.6% FLOPs, 66.1 at \(N=4\) with 25% FLOPs, and 65.9 at \(N=8\) with 23.6% FLOPs [2508.02134]. The paper describes \(N=2\) as the best trade-off in that setting and recommends, in its experiments, choosing

\[
N = \frac{\text{input frame number}}{64}.
\]

For temporal units \(M\), the overall score is reported as stable across \(M=1,4,32,64\), though larger \(M\) helps temporal perception and smaller \(M\) somewhat helps spatial perception [2508.02134]. For the reference fusion layer \(L\), the best setting shifts deeper as context length grows: for 128 frames with 50% drop, no fusion gives 65.8, \(L=1\) gives 65.4, and \(L=3\) gives 66.3; for 256 frames with 75% drop, no fusion gives 65.5, \(L=3\) gives 66.0, and \(L=6\) gives 66.3 [2508.02134].

## 6. Interpretation, limitations, and place in long-video Video-MLLM inference

Free-MoRef is presented as a method based on a particular reasoning hypothesis: many long-video questions do not require all visual tokens to interact densely with one another in every layer. Instead, what matters first is allowing the question to search broadly through the video for relevant clues, after which a smaller set of important visual tokens can be brought together for deeper joint reasoning [2508.02134]. Free-MoRef mirrors that pattern by splitting long context into multiple references, using question-guided fusion during MoRef-attention, and then performing a later reference fusion step.

The paper identifies several limitations and caveats. Before reference fusion, visual tokens from different references do not directly interact, so early cross-reference visual interactions are approximated rather than fully modeled [2508.02134]. Longer context can also hurt tasks whose evidence is highly localized; the paper specifically notes that Attribute Perception can worsen because extra video content introduces irrelevant information [2508.02134]. Performance depends on the hyperparameters \(N\), \(M\), and \(L\), which implies a tuning burden across backbones and task regimes [2508.02134]. Reference fusion is lossy because it prunes visual tokens, so if it is applied too early or too aggressively, useful information can be discarded [2508.02134]. Finally, although the method is argued to be generally applicable to Video-MLLMs whose bottleneck is LLM context length, the direct experiments reported are centered on LLaVA-Video-7B [2508.02134].

The same source also notes compatibility with FlashAttention and states that the method can be integrated with streaming inference or token compression if desired [2508.02134]. This suggests that Free-MoRef is best understood not as a replacement for all long-video techniques, but as an inference-path modification that can occupy an intermediate point between full long-context attention, compression-based reduction, and streaming-based serialization.

In that sense, Free-MoRef’s significance lies in redefining long-video inference as multiplexed short-context reasoning within a single inference pass. Its principal contribution is not a new pretrained architecture but a reorganization of existing Video-MLLM computation: multi-reference partition, query-aware MoRef-attention, and intermediate reference fusion [2508.02134]. On the evidence reported, this organization allows an existing Video-MLLM to process \(2\times\) to \(8\times\) longer frame inputs without compression, on a single A100 GPU, while improving benchmark performance over naive long-context inference and, in several comparisons, over dedicated long-video models [2508.02134].

Source: https://www.emergentmind.com/topics/free-moref