ReMoRa: Efficient Long-Video Understanding

• ReMoRa is a multimodal large language model for long-video understanding that processes compressed video streams to efficiently capture temporal dynamics.
• It refines codec-derived motion vectors through a dedicated module to achieve near-optical-flow fidelity while reducing computational cost.
• Hierarchical state-space compression fuses sparse appearance with dense motion features, enabling linear temporal modeling and superior benchmark performance.

ReMoRa is a multimodal LLM for long-video understanding that operates directly on compressed video representations rather than on a full stream of decoded RGB frames. Its design retains a sparse set of RGB keyframes for appearance, encodes temporal dynamics through motion representations derived from compressed video, introduces a refinement module to denoise and densify those motions, and compresses appearance-and-motion features so that temporal modeling scales linearly with sequence length. In the reported experiments, ReMoRa outperformed baseline methods on multiple long-video benchmarks, including LongVideoBench, NExT-QA, and MLVU (Yashima et al., 18 Feb 2026).

1. Problem setting and design objectives

ReMoRa is motivated by a specific systems bottleneck in video MLLMs: long-form video understanding becomes difficult when a model ingests full decoded frame sequences. A minute-long video at 16 fps is approximately 960 frames, and feeding all decoded RGB frames into a vision transformer plus an LLM incurs $O(T^2)$ self-attention cost while repeatedly re-encoding nearly identical background content. Standard uniform frame sampling is presented as an unsatisfactory compromise, because sparse sampling can miss short events and dense sampling becomes computationally intractable (Yashima et al., 18 Feb 2026).

The model is therefore organized around four explicit objectives. It avoids redundant full-frame encoding by operating directly on compressed video streams; retains appearance through a small set of I-frames; captures motion through block-based motion vectors; denoises and densifies those coarse codec motions to approach optical-flow fidelity; and compresses the resulting appearance-plus-motion features so that temporal modeling scales only linearly in video length. This design places ReMoRa within compressed-domain video understanding, but the distinguishing emphasis is its combination of codec-native motion extraction, a refinement stage for those motions, and a hierarchical state-space compression pipeline (Yashima et al., 18 Feb 2026).

2. Compressed-domain architecture

ReMoRa processes a video of $T$ frames by first re-encoding it into $K$ groups of pictures (GOPs). Each $GOP_k$ contains one I-frame $V^{(k,0)}$ and $T_g - 1$ P/B frames summarized by motion fields $\{m^{(k,t)}\}$. The pipeline begins with scene-adaptive GOP extraction, where the video is re-encoded with H.264 to insert I-frames at scene changes, with maximum GOP length $T_g = 32$ (Yashima et al., 18 Feb 2026).

Appearance processing is handled by an image encoder identified as SigLIP ViT-SO. Each I-frame $V^{(k,0)}$ is tiled into

$N_p = \frac{H \cdot W}{p^2}$

patches with $T$0 pixels, producing an embedding

$T$1

Temporal dynamics are encoded from compressed motion. Each block-level motion field

$T$2

is passed to the Refined Motion Representation module, which outputs

$T$3

Per GOP, the appearance and motion embeddings are concatenated as

$T$4

with

$T$5

This formulation makes the compressed stream—not the decoded RGB sequence—the primary visual substrate for the downstream multimodal model (Yashima et al., 18 Feb 2026).

3. Motion representation and refinement

The motion input to ReMoRa is derived from codec motion vectors. At encode time, each P/B frame is divided into macroblocks of size $T$6, specified here as $T$7. For a block index $T$8, the codec stores a two-dimensional displacement

$T$9

Collecting all blocks yields

$K$0

These vectors are described as a cheap, block-level proxy for optical flow, but also as spatially coarse, quantized, and noisy (Yashima et al., 18 Feb 2026).

The Refined Motion Representation (RMR) module is introduced to address precisely that fidelity gap. Its goal is to map coarse block motions to fine-grained dense flow features. The reported pretraining procedure prepares pseudo-ground-truth optical flow $K$1 via an off-the-shelf flow model, for example Co-Tracker3, and trains an RMR network—described as a small U-Net or light transformer—to take reshaped block motion input $K$2 and predict a dense motion estimate $K$3. The optimization objective is

$K$4

The summary also states the equivalent pretraining loss as $K$5, with an optional reconstruction loss $K$6 if residual reconstruction is used. During finetuning, the refined motion field is embedded into $K$7 (Yashima et al., 18 Feb 2026).

In the paper’s own interpretation, RMR “recovers near-optical-flow fidelity from block motions at low cost.” A plausible implication is that the compressed-domain representation is not treated merely as a storage convenience, but as a trainable surrogate modality whose deficiencies are explicitly corrected before multimodal fusion (Yashima et al., 18 Feb 2026).

4. Hierarchical Motion State Space compression

After per-GOP concatenation, ReMoRa applies a Hierarchical Motion State Space (HMSS) module to reduce the cost of long-range temporal modeling. The local stage uses a Mamba state-space model to fuse appearance and motion within each GOP and compress the sequence to a summary

$K$8

obtained by taking the first $K$9 tokens of the local scan. The global stage stacks the GOP summaries $GOP_k$0 and runs a bidirectional Mamba to produce the final sequence

$GOP_k$1

The two explicit transformations are

$GOP_k$2

and

$GOP_k$3

The stated complexity is $GOP_k$4, or effectively linear in $GOP_k$5, because the global stage operates on GOP-compressed summaries rather than on the full tokenized video stream (Yashima et al., 18 Feb 2026).

This compression strategy is central to the model’s claim of scalability. The paper contrasts it with naive flattening, which would require $GOP_k$6 attention cost, and presents HMSS as the mechanism that replaces full quadratic temporal attention with linear-scan state-space modeling. Within the reported ablations, HMSS also outperforms both cross-attention fusion and naive additive aggregation (Yashima et al., 18 Feb 2026).

5. Language-model interface and training regime

The final visual sequence $GOP_k$7 is projected into the LLM embedding space through a linear layer

$GOP_k$8

These projected visual tokens are concatenated with token embeddings

$GOP_k$9

and the combined sequence is fed to a pre-trained Qwen2 7B LLM equipped with LoRA adapters. The summary states that the LLM backbone is not changed beyond LoRA adapters on cross-attention layers, and decoding is autoregressive with standard cross-entropy loss on target tokens $V^{(k,0)}$0 (Yashima et al., 18 Feb 2026).

The training structure is therefore modular. The image encoder supplies appearance tokens; RMR is pretrained against pseudo-ground-truth optical flow; HMSS performs compressed temporal aggregation; and the LLM handles multimodal reasoning and answer generation. Because the architecture retains only scene-adaptive I-frames plus refined motion signals, the model is designed to preserve event-level temporal information without paying the cost of encoding every RGB frame. This suggests a division of labor in which appearance is sparsely refreshed while motion carries the dense temporal burden (Yashima et al., 18 Feb 2026).

6. Empirical performance, ablations, and limitations

On long-video benchmarks with a Qwen2-7B backbone, ReMoRa reports the following single-model numbers:

Benchmark	ReMoRa	Comparator note
LongVideoBench	60.8	2nd best 59.5
NExT-QA	84.2	2nd 83.2
MLVU	72.1	2nd 70.8
VideoMME	64.4	2nd 65.1
PerceptionTest	67.7	near-top
Average	69.8	best

On short-video QA, the reported numbers are MSVD-QA: $V^{(k,0)}$1, $V^{(k,0)}$2 and ActivityNet-QA: $V^{(k,0)}$3, $V^{(k,0)}$4 (Yashima et al., 18 Feb 2026).

The ablation studies isolate three design choices. For frame sampling, 64 scene-adaptive I-frames achieve $V^{(k,0)}$5 on VideoMME / NExT-QA, compared with $V^{(k,0)}$6 for 32 I-frames, $V^{(k,0)}$7 for 16 I-frames, and $V^{(k,0)}$8 for 64 uniform RGB frames. For motion refinement, the full system achieves $V^{(k,0)}$9, compared with $T_g - 1$0 without optical-flow pretraining and $T_g - 1$1 without RMR, using raw motion $T_g - 1$2. For GOP aggregation, HMSS achieves $T_g - 1$3, compared with $T_g - 1$4 for cross-attention fusion and $T_g - 1$5 for naive additive fusion (Yashima et al., 18 Feb 2026).

The limitations are also explicit. Spatial grounding errors remain, especially for object localization across frames. Some fine-grained motions still elude the RMR, with finger-level gestures given as the example. Annotation noise and question ambiguity can confound training. The future directions listed in the summary are integration of audio and subtitle streams, stronger spatial cross-attention, multi-scale motion encoders, end-to-end flow supervision, and further compression of appearance tokens. The applications named are video summarization, surveillance video QA, domestic robot scene understanding, and assistive technology for visually impaired users (Yashima et al., 18 Feb 2026).

A common source of confusion is the name itself. In arXiv usage, closely related spellings refer to unrelated systems: “Remora” also denotes a higher-order, rank-polymorphic array-processing programming language (Shivers et al., 2019, Slepak et al., 2019), “REMORA” denotes a UKRI-funded CubeSat-based asteroid-rendezvous programme (Soldini et al., 1 Jun 2026), and “Remora” denotes a compact direct-geometry time-of-flight spectrometer proposed for the ESS (Fogh et al., 13 Apr 2026). ReMoRa, by contrast, refers specifically to the long-video MLLM based on refined motion representation (Yashima et al., 18 Feb 2026).