Free-MoRef: Training-Free Long-Video Inference

• Free-MoRef is a method that reconstructs lengthy video token sequences into multiple shorter 'references' to enable parallel processing in a single inference pass.
• It leverages a modified MoRef-attention mechanism to fuse query tokens across references, balancing efficiency with rich cross-reference interactions without additional training.
• By reducing attention complexity and enabling efficient token fusion, Free-MoRef processes 2x–8x longer frame inputs on a single GPU while delivering improved benchmark performance.

Searching arXiv for the Free-MoRef paper and closely related long-video Video-MLLM context. Free-MoRef is a training-free inference method for long-video understanding in Video Multimodal LLMs (Video-MLLMs). It is designed to extend effective context perception within a single forward pass by reconstructing an overlong visual token sequence into several shorter “references,” processing those references in parallel with a modified attention mechanism called MoRef-attention, and then fusing the references at an intermediate decoder layer so that some cross-reference interactions can be recovered before generation continues (Wang et al., 4 Aug 2025). In the formulation reported for LLaVA-Video-7B, Free-MoRef is presented as an alternative to token compression and streaming inference, with the stated aim of achieving fuller perception of more frames without compression while retaining the feel of instant response because everything is handled in one inference (Wang et al., 4 Aug 2025).

1. Problem setting and conceptual role

Free-MoRef is motivated by a central systems bottleneck in Video-MLLMs: videos generate substantially more tokens than images, while the underlying decoder-only LLM remains constrained by a practical context limit and by the quadratic cost of self-attention (Wang et al., 4 Aug 2025). The reported example is LLaVA-Video with Qwen2 as the LLM, where a threshold around $32{,}768$ tokens is described as the point for stable inference; the default setup uses up to $64$ frames, each represented by $182$ tokens, or about $11{,}648$ visual tokens before adding text, whereas $256$ or $512$ frames would push the token count toward about $46\text{k}$ or $93\text{k}+$ (Wang et al., 4 Aug 2025).

Within that setting, Free-MoRef is positioned against two dominant solution families. Token compression reduces the number of visual tokens by pruning or summarization, but the trade-off is loss of feature granularity and information fidelity. Streaming inference preserves detail by processing the video in segments while reusing historical information such as KV-cache, but latency scales with context length because the model must process the video in multiple stages (Wang et al., 4 Aug 2025). Free-MoRef is framed as a third option: instead of compressing a long video into fewer tokens or serializing chunk processing across time, it creates multiple parallel short-context views of the long video and allows the same question token to query all of them at once (Wang et al., 4 Aug 2025).

The method is described as “training-free,” meaning that it requires no retraining of the base Video-MLLM, no finetuning to support longer contexts, and no learned routing or fusion parameters specific to the method. What changes is the organization of inference: the visual tokens are repartitioned, attention is modified across references, and hidden activations are fused at an intermediate stage (Wang et al., 4 Aug 2025). This makes the method immediately deployable on an existing Video-MLLM in the sense claimed by the paper.

The name “Free-MoRef” is explicitly interpreted as a training-free Mixture-of-References inference method. The paper states that it is inspired by Mixture-of-Experts, but the analogy is limited: there are no separately trained expert subnetworks, all references share the same model weights, and what differs across branches is the context slice each branch sees rather than the parameters themselves (Wang et al., 4 Aug 2025). This suggests that the central novelty lies in input-side multiplexing of context perception rather than parameter-side specialization.

2. Multi-reference partition and inference organization

The first stage of Free-MoRef is Multi-Reference Partition. An overlong visual token sequence is reconstructed into $N$ shorter sequences called references or chunks (Wang et al., 4 Aug 2025). Each reference contains the same system prompt, a subset or reconstruction of visual tokens, and the same question. The references are intended to be partial but informative abstractions of the full long video (Wang et al., 4 Aug 2025).

The partition is controlled by two hyperparameters. The whole visual token sequence is first divided temporally into $M$ units, and each unit is then decomposed into $64$0 fragments. Fragments from different units are aggregated to form $64$1 reference chunks (Wang et al., 4 Aug 2025). The procedure is described in the data as follows:

$64$2

This reconstructed expression is explicitly identified in the source material as a faithful reconstruction from the text rather than a printed equation. Its significance is architectural: each reference can contain tokens drawn from across the full video timeline rather than from one contiguous segment, depending on $64$3 (Wang et al., 4 Aug 2025). The paper interprets $64$4 as temporally independent large chunks, whereas larger $64$5 creates stronger temporal interleaving across references (Wang et al., 4 Aug 2025).

Each reference sequence is written as

$64$6

and stacking all references yields

$64$7

where $64$8 is the number of references, $64$9 is the per-reference sequence length, and $182$0 is the embedding dimension (Wang et al., 4 Aug 2025). All chunks share the same system and question tokens; only the visual tokens differ.

The operational claim made for this organization is that the same base model that can reason over one short sequence is repurposed to reason over several short sequences in parallel, after which their evidence is merged (Wang et al., 4 Aug 2025). This is the sense in which Free-MoRef is said to “instantly multiplex” context perception capabilities.

3. MoRef-attention and reference fusion

MoRef-attention is the core mechanism of the method. The paper states that the LLM’s self-attention is modified so that multiple references are processed in parallel, while the question-token updates are fused across references (Wang et al., 4 Aug 2025). The tensors are given as

$182$1

and standard attention is run within each chunk, producing outputs decomposed as

$182$2

Because the system prompt is identical in all references, $182$3 is reported as identical across chunks, whereas $182$4 and $182$5 differ because the visual tokens differ (Wang et al., 4 Aug 2025).

The defining operation is fusion of question outputs across references:

$182$6

The fused query activation is copied back to every chunk, so subsequent layers begin from the same query representation in all branches (Wang et al., 4 Aug 2025). The final output of MoRef-attention is then

$182$7

This means that the system tokens remain unchanged, the vision hidden states remain chunk-specific, and the question hidden states become globally mixed across references (Wang et al., 4 Aug 2025).

The weights $182$8 are derived from an auxiliary cross-modal attention map between question queries and vision keys:

$182$9

Because the implementation is stated to use FlashAttention, which does not directly expose attention weights, this auxiliary map is computed manually (Wang et al., 4 Aug 2025). The reference weight for chunk $11{,}648$0 is then

$11{,}648$1

Thus, each chunk contributes to the fused question activation in proportion to its strongest query-to-vision match (Wang et al., 4 Aug 2025).

The method also introduces a later Reference Fusion stage. The paper states that after the shadow layers in the LLM, attention becomes more concentrated on query tokens, and on that basis the model stops parallel multi-reference processing at an intermediate decoder layer $11{,}648$2, selects important visual tokens from each chunk, merges them into a single global sequence, and continues the remaining decoder layers with standard attention (Wang et al., 4 Aug 2025). The term “shadow layers” is described in the source summary as likely referring operationally to shallow early decoder layers; this interpretation is explicitly marked as explanatory rather than as a verbatim statement from the paper.

For reference fusion, the same auxiliary attention map is used, now with shape

$11{,}648$3

and averaging over the question dimension yields a vision-token importance matrix

$11{,}648$4

The summary reconstructs the implied formula as

$11{,}648$5

again explicitly as a faithful reconstruction from the text (Wang et al., 4 Aug 2025). Based on $11{,}648$6, the least important $11{,}648$7 fraction of visual tokens in each chunk is pruned, so that approximately a $11{,}648$8 fraction survives from each reference (Wang et al., 4 Aug 2025). The retained tokens are then merged in temporal order into a global reference, summarized in the source as

$11{,}648$9

The reason given for this step is that MoRef-attention does not model direct interactions among visual tokens from different references in shallow layers, whereas reference fusion brings key visual tokens into one sequence so that ordinary self-attention in later layers can capture cross-reference dependencies (Wang et al., 4 Aug 2025).

4. Computational properties and scaling behavior

The paper argues that Free-MoRef reduces the effective attention burden by partitioning a long visual sequence into $256$0 non-overlapping chunks, which lowers attention complexity by approximately a factor of

$256$1

relative to full attention over the whole sequence (Wang et al., 4 Aug 2025). The stated intuition is the usual one: if full attention over sequence length $256$2 scales like $256$3, then $256$4 chunkwise attention computations over length $256$5 cost $256$6 (Wang et al., 4 Aug 2025).

This scaling argument is paired with concrete FLOP comparisons for LLaVA-Video-7B. Relative to the 64-frame baseline at 100% FLOPs, the paper reports the following values (Wang et al., 4 Aug 2025):

Setting	FLOPs
64 frames baseline	100%
128 frames naive	400%
128 frames + Free-MoRef	110.4%
256 frames naive	1600%
256 frames + Free-MoRef	163.2%
512 frames naive	6400%
512 frames + Free-MoRef	400%

An ablation for 128 frames with $256$7 reports 27.6% FLOPs under the chunked setting used in that experiment (Wang et al., 4 Aug 2025). The paper interprets these results as evidence that naive long-context scaling is prohibitive, whereas chunked parallel references keep the growth much milder.

The system claim attached to these efficiency numbers is that Free-MoRef can achieve full perception of $256$8 to $256$9 longer input frames without compression on a single A100 GPU while keeping instant responses (Wang et al., 4 Aug 2025). The conclusion also states that the method reached comprehensive understanding of 1024 frames on a 7B Video-LLM with one A100 (Wang et al., 4 Aug 2025). The term “instant” is used comparatively, to distinguish this single-pass parallel reasoning regime from streaming inference, in which longer context implies additional serial stages and correspondingly larger first-token latency (Wang et al., 4 Aug 2025).

5. Experimental results and empirical profile

The reported empirical evaluation uses LLaVA-Video-7B on VideoMME, MLVU, and LongVideoBench, with lmms-eval on a single A100 GPU (Wang et al., 4 Aug 2025). The default frame-loading setup is FPS $512$0, up to 64 frames, and 182 tokens per frame. Extended settings use 128, 256, and 512 frames; the paper reports $512$1, $512$2 for the $512$3, $512$4, and $512$5 settings respectively, with reference fusion layers $512$6 (Wang et al., 4 Aug 2025).

For the main long-context study on LLaVA-Video-7B, the reported benchmark results are as follows (Wang et al., 4 Aug 2025):

Setting	MLVU	VideoMME overall	LongVideoBench overall
64 frames baseline	70.3	64.3	58.8
128 frames naive	70.2	64.9	58.7
128 frames + Free-MoRef	70.8	66.3	59.3
256 frames naive	67.2	63.1	56.7
256 frames + Free-MoRef	72.5	66.3	59.3
512 frames naive	61.1	60.6	54.3
512 frames + Free-MoRef	72.8	66.9	59.9

These numbers are central to the paper’s claim. Naively increasing frames provides little gain at 128 frames and degraded performance at 256 and 512 frames, whereas Free-MoRef improves performance as context length increases because, on the paper’s interpretation, it can exploit the additional information without overwhelming the model (Wang et al., 4 Aug 2025).

The comparison table against other models around the 7B-8B scale reports the following headline numbers: InternVL2 at 64.0 / 54.6 / 54.0 on MLVU / LongVideoBench / VideoMME overall; InternVL2.5 at 68.4 / 57.5 / 64.5; Qwen2-VL at 64.8 / 55.6 / 63.3; LLaVA-OneVision at 64.7 / 56.3 / 58.2; LLaVA-Video at 70.2 / 58.2 / 64.3; Kangaroo at 61.0 / 54.8 / 56.0; LongVILA at – / 57.1 / 60.1; LongVA at 56.3 / – / 52.6; Video-XL at 64.9 / 50.7 / 55.5; RETAKE at 69.8 / – / 63.9; and LLaVA-Video with Free-MoRef at 72.8 / 59.9 / 66.9 (Wang et al., 4 Aug 2025). On the VideoMME long subset, the paper notes that Free-MoRef reaches 56.0, slightly below RETAKE’s 56.2, while still giving the best overall model on VideoMME overall and the highest scores in the table on MLVU and LongVideoBench (Wang et al., 4 Aug 2025).

The ablation studies are also structurally important. The component ablation reports 64.9 for the 128-frame setting without partition, MoRef, or fusion; 63.9 for fusion only; 62.0 for partition with full-attention-separate and fusion but no MoRef; 65.8 for partition with MoRef and no fusion; and 66.3 for partition with MoRef and fusion (Wang et al., 4 Aug 2025). This shows that partitioning alone does not account for the performance gain; the performance increase is attributed primarily to MoRef-attention, with reference fusion adding a further boost (Wang et al., 4 Aug 2025).

The study of the number of references $512$7 at 128 frames reports 64.9 at $512$8 with 100% FLOPs, 66.3 at $512$9 with 27.6% FLOPs, 66.1 at $46\text{k}$0 with 25% FLOPs, and 65.9 at $46\text{k}$1 with 23.6% FLOPs (Wang et al., 4 Aug 2025). The paper describes $46\text{k}$2 as the best trade-off in that setting and recommends, in its experiments, choosing

$46\text{k}$3

For temporal units $46\text{k}$4, the overall score is reported as stable across $46\text{k}$5, though larger $46\text{k}$6 helps temporal perception and smaller $46\text{k}$7 somewhat helps spatial perception (Wang et al., 4 Aug 2025). For the reference fusion layer $46\text{k}$8, the best setting shifts deeper as context length grows: for 128 frames with 50% drop, no fusion gives 65.8, $46\text{k}$9 gives 65.4, and $93\text{k}+$0 gives 66.3; for 256 frames with 75% drop, no fusion gives 65.5, $93\text{k}+$1 gives 66.0, and $93\text{k}+$2 gives 66.3 (Wang et al., 4 Aug 2025).

6. Interpretation, limitations, and place in long-video Video-MLLM inference

Free-MoRef is presented as a method based on a particular reasoning hypothesis: many long-video questions do not require all visual tokens to interact densely with one another in every layer. Instead, what matters first is allowing the question to search broadly through the video for relevant clues, after which a smaller set of important visual tokens can be brought together for deeper joint reasoning (Wang et al., 4 Aug 2025). Free-MoRef mirrors that pattern by splitting long context into multiple references, using question-guided fusion during MoRef-attention, and then performing a later reference fusion step.

The paper identifies several limitations and caveats. Before reference fusion, visual tokens from different references do not directly interact, so early cross-reference visual interactions are approximated rather than fully modeled (Wang et al., 4 Aug 2025). Longer context can also hurt tasks whose evidence is highly localized; the paper specifically notes that Attribute Perception can worsen because extra video content introduces irrelevant information (Wang et al., 4 Aug 2025). Performance depends on the hyperparameters $93\text{k}+$3, $93\text{k}+$4, and $93\text{k}+$5, which implies a tuning burden across backbones and task regimes (Wang et al., 4 Aug 2025). Reference fusion is lossy because it prunes visual tokens, so if it is applied too early or too aggressively, useful information can be discarded (Wang et al., 4 Aug 2025). Finally, although the method is argued to be generally applicable to Video-MLLMs whose bottleneck is LLM context length, the direct experiments reported are centered on LLaVA-Video-7B (Wang et al., 4 Aug 2025).

The same source also notes compatibility with FlashAttention and states that the method can be integrated with streaming inference or token compression if desired (Wang et al., 4 Aug 2025). This suggests that Free-MoRef is best understood not as a replacement for all long-video techniques, but as an inference-path modification that can occupy an intermediate point between full long-context attention, compression-based reduction, and streaming-based serialization.

In that sense, Free-MoRef’s significance lies in redefining long-video inference as multiplexed short-context reasoning within a single inference pass. Its principal contribution is not a new pretrained architecture but a reorganization of existing Video-MLLM computation: multi-reference partition, query-aware MoRef-attention, and intermediate reference fusion (Wang et al., 4 Aug 2025). On the evidence reported, this organization allows an existing Video-MLLM to process $93\text{k}+$6 to $93\text{k}+$7 longer frame inputs without compression, on a single A100 GPU, while improving benchmark performance over naive long-context inference and, in several comparisons, over dedicated long-video models (Wang et al., 4 Aug 2025).