---
title: 'ReMoRa: Efficient Long-Video Understanding'
url: https://www.emergentmind.com/topics/remora
type: topic
---

# ReMoRa: Efficient Long-Video Understanding

ReMoRa is a multimodal large language model for long-video understanding that operates directly on compressed video representations rather than on a full stream of decoded RGB frames. Its design retains a sparse set of RGB keyframes for appearance, encodes temporal dynamics through motion representations derived from compressed video, introduces a refinement module to denoise and densify those motions, and compresses appearance-and-motion features so that temporal modeling scales linearly with sequence length. In the reported experiments, ReMoRa outperformed baseline methods on multiple long-video benchmarks, including LongVideoBench, NExT-QA, and MLVU [2602.16412].

## 1. Problem setting and design objectives

ReMoRa is motivated by a specific systems bottleneck in video MLLMs: long-form video understanding becomes difficult when a model ingests full decoded frame sequences. A minute-long video at 16 fps is approximately 960 frames, and feeding all decoded RGB frames into a vision transformer plus an LLM incurs \(O(T^2)\) self-attention cost while repeatedly re-encoding nearly identical background content. Standard uniform frame sampling is presented as an unsatisfactory compromise, because sparse sampling can miss short events and dense sampling becomes computationally intractable [2602.16412].

The model is therefore organized around four explicit objectives. It avoids redundant full-frame encoding by operating directly on compressed video streams; retains appearance through a small set of I-frames; captures motion through block-based motion vectors; denoises and densifies those coarse codec motions to approach optical-flow fidelity; and compresses the resulting appearance-plus-motion features so that temporal modeling scales only linearly in video length. This design places ReMoRa within compressed-domain video understanding, but the distinguishing emphasis is its combination of codec-native motion extraction, a refinement stage for those motions, and a hierarchical state-space compression pipeline [2602.16412].

## 2. Compressed-domain architecture

ReMoRa processes a video of \(T\) frames by first re-encoding it into \(K\) groups of pictures (GOPs). Each \(GOP_k\) contains one I-frame \(V^{(k,0)}\) and \(T_g - 1\) P/B frames summarized by motion fields \(\{m^{(k,t)}\}\). The pipeline begins with scene-adaptive GOP extraction, where the video is re-encoded with H.264 to insert I-frames at scene changes, with maximum GOP length \(T_g = 32\) [2602.16412].

Appearance processing is handled by an image encoder identified as SigLIP ViT-SO. Each I-frame \(V^{(k,0)}\) is tiled into
\[
N_p = \frac{H \cdot W}{p^2}
\]
patches with \(p = 16\) pixels, producing an embedding
\[
e_I^{(k)} \in \mathbb{R}^{N_p \times d_s}.
\]
Temporal dynamics are encoded from compressed motion. Each block-level motion field
\[
m^{(k,t)} \in \mathbb{R}^{N_m \times 2}
\]
is passed to the Refined Motion Representation module, which outputs
\[
E_M^{(k,t)} \in \mathbb{R}^{N_m \times d_s}.
\]
Per GOP, the appearance and motion embeddings are concatenated as
\[
Z^{(k)} = [E_I^{(k)}; E_M^{(k,1)}; \ldots; E_M^{(k,T_g-1)}] \in \mathbb{R}^{L_g \times d_s},
\]
with
\[
L_g = N_p + (T_g - 1)N_m.
\]
This formulation makes the compressed stream—not the decoded RGB sequence—the primary visual substrate for the downstream multimodal model [2602.16412].

## 3. Motion representation and refinement

The motion input to ReMoRa is derived from codec motion vectors. At encode time, each P/B frame is divided into macroblocks of size \(b_h \times b_w\), specified here as \(4 \times 4\). For a block index \(i = (u,v)\), the codec stores a two-dimensional displacement
\[
m_i^{(k,t)} = (\Delta x_i, \Delta y_i).
\]
Collecting all blocks yields
\[
m^{(k,t)} \in \mathbb{R}^{N_m \times 2}, \qquad N_m = \frac{H}{b_h}\cdot\frac{W}{b_w}.
\]
These vectors are described as a cheap, block-level proxy for optical flow, but also as spatially coarse, quantized, and noisy [2602.16412].

The Refined Motion Representation (RMR) module is introduced to address precisely that fidelity gap. Its goal is to map coarse block motions to fine-grained dense flow features. The reported pretraining procedure prepares pseudo-ground-truth optical flow \(F^{(k,t)}\) via an off-the-shelf flow model, for example Co-Tracker3, and trains an RMR network—described as a small U-Net or light transformer—to take reshaped block motion input \(m^{(k,t)} \in \mathbb{R}^{h \times w \times 2}\) and predict a dense motion estimate \(\hat{W}^{(k,t)} \in \mathbb{R}^{h' \times w' \times 2}\). The optimization objective is
\[
L_{\text{flow}} = \left\|\hat{W}^{(k,t)} - F^{(k,t)}\right\|_2^2.
\]
The summary also states the equivalent pretraining loss as \(L_{\text{motion}} = L2(F_{\text{pred}}, F_{\text{target}})\), with an optional reconstruction loss \(L_{\text{rec}}\) if residual reconstruction is used. During finetuning, the refined motion field is embedded into \(E_M^{(k,t)} \in \mathbb{R}^{N_m \times d_s}\) [2602.16412].

In the paper’s own interpretation, RMR “recovers near-optical-flow fidelity from block motions at low cost.” A plausible implication is that the compressed-domain representation is not treated merely as a storage convenience, but as a trainable surrogate modality whose deficiencies are explicitly corrected before multimodal fusion [2602.16412].

## 4. Hierarchical Motion State Space compression

After per-GOP concatenation, ReMoRa applies a Hierarchical Motion State Space (HMSS) module to reduce the cost of long-range temporal modeling. The local stage uses a Mamba state-space model to fuse appearance and motion within each GOP and compress the sequence to a summary
\[
h_{\text{gop}}^{(k)} \in \mathbb{R}^{N_p \times d_s},
\]
obtained by taking the first \(N_p\) tokens of the local scan. The global stage stacks the GOP summaries \(\{h_{\text{gop}}^{(0)}, \ldots, h_{\text{gop}}^{(K-1)}\}\) and runs a bidirectional Mamba to produce the final sequence
\[
H \in \mathbb{R}^{K \cdot N_p \times d_s}.
\]
The two explicit transformations are
\[
Z^{(k)} \rightarrow \text{SSM}_{\text{local}} \rightarrow Z_I^{(k)} = \text{SSM}_{\text{local}}(Z^{(k)})[1:N_p] \in \mathbb{R}^{N_p \times d_s},
\]
and
\[
[Z_I^{(0)}; \ldots; Z_I^{(K-1)}] \rightarrow \text{SSM}_{\text{global}} \rightarrow H \in \mathbb{R}^{K \cdot N_p \times d_s}.
\]
The stated complexity is \(O(K \cdot L_g + K^2)\), or effectively linear in \(K \cdot L_g\), because the global stage operates on GOP-compressed summaries rather than on the full tokenized video stream [2602.16412].

This compression strategy is central to the model’s claim of scalability. The paper contrasts it with naive flattening, which would require \(O((K \cdot L_g)^2)\) attention cost, and presents HMSS as the mechanism that replaces full quadratic temporal attention with linear-scan state-space modeling. Within the reported ablations, HMSS also outperforms both cross-attention fusion and naive additive aggregation [2602.16412].

## 5. Language-model interface and training regime

The final visual sequence \(H\) is projected into the LLM embedding space through a linear layer
\[
W_{\text{proj}} \in \mathbb{R}^{d_s \times d_{\text{LLM}}}.
\]
These projected visual tokens are concatenated with token embeddings
\[
x_{\text{txt}} \in \mathbb{R}^{L_{\text{txt}} \times d_{\text{LLM}}},
\]
and the combined sequence is fed to a pre-trained Qwen2 7B LLM equipped with LoRA adapters. The summary states that the LLM backbone is not changed beyond LoRA adapters on cross-attention layers, and decoding is autoregressive with standard cross-entropy loss on target tokens \(y\) [2602.16412].

The training structure is therefore modular. The image encoder supplies appearance tokens; RMR is pretrained against pseudo-ground-truth optical flow; HMSS performs compressed temporal aggregation; and the language model handles multimodal reasoning and answer generation. Because the architecture retains only scene-adaptive I-frames plus refined motion signals, the model is designed to preserve event-level temporal information without paying the cost of encoding every RGB frame. This suggests a division of labor in which appearance is sparsely refreshed while motion carries the dense temporal burden [2602.16412].

## 6. Empirical performance, ablations, and limitations

On long-video benchmarks with a Qwen2-7B backbone, ReMoRa reports the following single-model numbers:

| Benchmark | ReMoRa | Comparator note |
|---|---:|---|
| LongVideoBench | 60.8 | 2nd best 59.5 |
| NExT-QA | 84.2 | 2nd 83.2 |
| MLVU | 72.1 | 2nd 70.8 |
| VideoMME | 64.4 | 2nd 65.1 |
| PerceptionTest | 67.7 | near-top |
| Average | 69.8 | best |

On short-video QA, the reported numbers are MSVD-QA: \(Acc = 73.1\), \(Score = 4.0\) and ActivityNet-QA: \(Acc = 60.5\), \(Score = 3.7\) [2602.16412].

The ablation studies isolate three design choices. For frame sampling, 64 scene-adaptive I-frames achieve \(64.3 / 84.2\) on VideoMME / NExT-QA, compared with \(61.9 / 82.7\) for 32 I-frames, \(58.5 / 81.6\) for 16 I-frames, and \(62.4 / 82.8\) for 64 uniform RGB frames. For motion refinement, the full system achieves \(64.3 / 84.2\), compared with \(63.4 / 82.2\) without optical-flow pretraining and \(62.1 / 82.0\) without RMR, using raw motion \(m\). For GOP aggregation, HMSS achieves \(64.3 / 84.2\), compared with \(62.5 / 81.9\) for cross-attention fusion and \(61.3 / 81.5\) for naive additive fusion [2602.16412].

The limitations are also explicit. Spatial grounding errors remain, especially for object localization across frames. Some fine-grained motions still elude the RMR, with finger-level gestures given as the example. Annotation noise and question ambiguity can confound training. The future directions listed in the summary are integration of audio and subtitle streams, stronger spatial cross-attention, multi-scale motion encoders, end-to-end flow supervision, and further compression of appearance tokens. The applications named are video summarization, surveillance video QA, domestic robot scene understanding, and assistive technology for visually impaired users [2602.16412].

A common source of confusion is the name itself. In arXiv usage, closely related spellings refer to unrelated systems: “Remora” also denotes a higher-order, rank-polymorphic array-processing programming language [1912.13451; 1907.00509], “REMORA” denotes a UKRI-funded CubeSat-based asteroid-rendezvous programme [2606.02665], and “Remora” denotes a compact direct-geometry time-of-flight spectrometer proposed for the ESS [2604.11383]. ReMoRa, by contrast, refers specifically to the long-video MLLM based on refined motion representation [2602.16412].

Source: https://www.emergentmind.com/topics/remora