Gemini‑MVR: 3D Motion Video Retriever

• Gemini‑MVR is a dual-encoder retrieval system that converts text motion descriptions into human-centric video priors from large-scale databases.
• It employs separate action-level and object-level branches to capture fine-grained human motion and semantic context respectively.
• The system mitigates out-of-domain and vocabulary issues in motion LLMs, leading to significant improvements in 3D motion generation metrics.

The Gemini Motion Video Retriever (Gemini‑MVR) is the retrieval component of VimoRAG, a video-based retrieval-augmented framework for 3D motion generation with motion LLMs. In VimoRAG, Gemini‑MVR converts a text motion description into a relevant in-the-wild video from the Human-centric Video Database (HcVD), so that the downstream motion LLM can condition on retrieved 2D human motion signals rather than relying on text alone. Its stated purpose is to alleviate severe out-of-domain and out-of-vocabulary failure modes in motion LLMs trained on small text–motion corpora such as HumanML3D, which contains only about 14k text–motion pairs (Xu et al., 16 Aug 2025).

1. Functional role and problem setting

Within VimoRAG, Gemini‑MVR is the front-end retriever in a two-stage pipeline. Given a motion description $t$, it searches HcVD and returns the best matching video $v^\*$, specifically the rank-1 retrieval used in the paper. The motion LLM, implemented with Phi‑3‑3.8B plus a visual adapter, then consumes both the text prompt and the retrieved video, encoded into segment-wise embeddings $E^v$, and generates discrete motion tokens $y=\{y_1,\dots,y_{n_y}\}$, which are finally decoded by a VQ‑VAE into a 3D motion sequence (Xu et al., 16 Aug 2025).

The retriever is motivated by a specific limitation of existing motion LLMs. Systems such as MotionGPT are trained on relatively small text–motion datasets and therefore encounter two failure classes: out-of-domain prompts describing actions or styles absent from training, and out-of-vocabulary prompts involving unseen words or long composite descriptions. VimoRAG treats large-scale video corpora as a source of motion priors, but it also argues that video-based motion retrieval is nontrivial because generic video foundation models excel at object and scene recognition yet struggle with fine-grained human poses and behavior-only queries, particularly when object or environment cues are weak (Xu et al., 16 Aug 2025).

A common misconception is that generic text-to-video retrieval is sufficient for motion augmentation. The VimoRAG formulation rejects that premise explicitly. It frames motion retrieval as requiring human-centric matching of descriptions such as “running sideways while looking back” or “seated chopping wood, lift and strike,” and it further emphasizes robustness to partially mismatched retrievals because retrieval errors can propagate into the generator unless the model learns to discount them (Xu et al., 16 Aug 2025).

2. Dual-retriever architecture and motion representations

Gemini‑MVR is built as a CLIP-style dual-encoder retrieval system with two independent channels: an object-level retriever and an action-level retriever. The object-level branch uses InternVideo-derived text and video encoders, denoted $\theta_{\mathcal G}$ and $\theta_{\mathcal O}$, to represent generic semantic content. The action-level branch uses a text encoder $\theta_{\mathcal P}$ together with a keypoint-based video encoder $\theta_{\mathcal A}$, initialized from AlphaPose for 2D keypoint detection, MotionBERT as the keypoint encoder backbone, and a temporal Transformer (Xu et al., 16 Aug 2025).

The action branch represents a video $v$ with $m$ frames as a sequence of 2D body keypoints,

$v^\*$0

Each frame is encoded by MotionBERT and projected into a 768-dimensional feature,

$v^\*$1

After adding learnable positional embeddings $v^\*$2, the sequence $v^\*$3 is passed through a temporal Transformer with 4 layers, 12 heads, and width 768, yielding framewise outputs $v^\*$4. Mean pooling then forms the video-level action embedding,

$v^\*$5

This $v^\*$6 is the action representation used for similarity computation (Xu et al., 16 Aug 2025).

Text is encoded twice. The predicate-semantic representation is

$v^\*$7

and the argument-semantic representation is

$v^\*$8

On the video side, the object-level embedding is

$v^\*$9

The paper makes an important qualification: despite the names “predicate” and “argument,” these text encoders do not hard-code predicate/argument parsing. Their specialization emerges implicitly through contrastive alignment with different video features (Xu et al., 16 Aug 2025).

This architecture is designed to separate human action cues from object cues. When a query is dominated by verbs, body parts, and temporal evolution, the action branch is intended to dominate; when object and environment semantics are informative, the object branch can contribute more strongly. The resulting system is therefore neither a pure keypoint retriever nor a pure foundation-model retriever.

3. Contrastive learning and keypoints-aware similarity routing

The action-level retriever is trained with CLIP-style contrastive learning over text–video pairs. For a batch of size $E^v$0, with action-text embeddings $E^v$1 and action-video embeddings $E^v$2, cosine similarity is used: $E^v$3 The training objective is symmetric, consisting of text-to-action and action-to-text contrastive losses: $E^v$4 The object-level retriever is trained analogously with $E^v$5 and $E^v$6, yielding

$E^v$7

These two branches are trained independently in the first training stage (Xu et al., 16 Aug 2025).

The distinctive component is the action-aware similarity integrator $E^v$8, described as a lightweight integrator and also characterized as a keypoints-aware router. It receives the action embedding $E^v$9 and outputs two scalars $y=\{y_1,\dots,y_{n_y}\}$0, which softly gate the action-level and object-level similarities. The final retrieval score is

$y=\{y_1,\dots,y_{n_y}\}$1

The integrator is itself trained with a symmetric CLIP-style text–video contrastive loss,

$y=\{y_1,\dots,y_{n_y}\}$2

At inference, Gemini‑MVR computes $y=\{y_1,\dots,y_{n_y}\}$3 for all videos in HcVD, ranks them, and selects the top-1 video $y=\{y_1,\dots,y_{n_y}\}$4 (Xu et al., 16 Aug 2025).

This design directly addresses noisy retrieval. Rather than fixing a static weighting between action and object similarity, Gemini‑MVR conditions the fusion on the action embedding itself. The paper’s claim is that videos with more prominent human motion signals receive higher action weight, which improves retrieval quality and reduces downstream error propagation (Xu et al., 16 Aug 2025).

4. Conditioning the motion LLM and interaction with McDPO

Once Gemini‑MVR retrieves $y=\{y_1,\dots,y_{n_y}\}$5, the downstream generator treats the video as a prior rather than as an obligatory template. The retrieved video is encoded by InternVideo2 into segment-wise embeddings

$y=\{y_1,\dots,y_{n_y}\}$6

which are combined with the system prompt $y=\{y_1,\dots,y_{n_y}\}$7, instruction template $y=\{y_1,\dots,y_{n_y}\}$8, and text input $y=\{y_1,\dots,y_{n_y}\}$9 to form

$\theta_{\mathcal G}$0

The instruction format used in the paper is: “Generate a sequence of motion tokens matching the following human motion description. You can use the video as a reference. Video information: {Retrieved Video v} Motion description: {Input Text x}” (Xu et al., 16 Aug 2025).

During supervised fine-tuning, the target 3D motion is quantized by a VQ‑VAE into discrete tokens $\theta_{\mathcal G}$1, and the model is trained with cross-entropy,

$\theta_{\mathcal G}$2

At inference, the model autoregressively samples $\theta_{\mathcal G}$3, which the VQ‑VAE decodes into 3D skeleton trajectories with 22 joints and a 263-dimensional feature representation (Xu et al., 16 Aug 2025).

The retriever’s role is inseparable from the Motion-centric Dual-alignment DPO Trainer (McDPO), even though McDPO optimizes the generator rather than the retriever. McDPO defines a reward that combines motion–motion alignment and text–motion alignment: $\theta_{\mathcal G}$4 with $\theta_{\mathcal G}$5 and $\theta_{\mathcal G}$6. Chosen and rejected generations are then used in a DPO objective with $\theta_{\mathcal G}$7 to fine-tune the policy against a reference model (Xu et al., 16 Aug 2025).

This interaction clarifies another misconception: the retrieved video is not meant to be followed blindly. The reward is explicitly described as agnostic to video quality. If Gemini‑MVR returns a poor video, McDPO still rewards generations that match the text and the target motion, thereby teaching the motion LLM when to trust the retrieved prior and when to ignore it. The paper reports that McDPO-trained models are robust even when the video is random, as seen in the Mc‑R∞ versus NMc‑R∞ comparison (Xu et al., 16 Aug 2025).

5. Training corpus and empirical performance

Gemini‑MVR is trained on the Human-centric Video Database (HcVD), which contains 425,988 videos drawn from MotionX (32,500), UCF101 (13,320), NTU RGB+D (114,480), ASLAN (3,697), HMDB51 (6,849), Kinetics‑400 (306,245), and PennAction (2,326). Although VimoRAG is intended to operate without annotated captions during the RAG pipeline, the retriever itself is trained using synthetic captions generated by Qwen2‑VL‑7B‑Instruct with the prompt: “Please describe the person's actions in the video using a single sentence that contains a series of verbs.” One synthetic caption is generated per video. To enforce human-centricity, AlphaPose is used to detect humans and videos without human detections are filtered out (Xu et al., 16 Aug 2025).

On HcVD test splits, Gemini‑MVR improves text-to-video retrieval over InternVideo. On the human-centric video set with pool size 1990, R@1 improves from 53.6 to 58.3, R@5 from 84.5 to 87.3, R@10 from 92.3 to 93.7, and MnR from 4.2 to 3.6. On the single human-centric video set, R@1 improves from 52.3 to 61.0, R@5 from 84.0 to 89.2, R@10 from 91.5 to 94.1, and MnR from 4.5 to 3.5. These results are presented as evidence that keypoint modeling and dual routing improve motion-focused retrieval quality (Xu et al., 16 Aug 2025).

The downstream effect on 3D motion generation is substantial. On IDEA400 (OOD), VimoRAG with Gemini‑MVR and McDPO achieves FID 2.388, compared with MotionGPT’s 5.544, while also improving R-Precision Top1/Top2/Top3 with diversity comparable or better. On HumanML3D, using the same Phi‑3‑3.8B backbone, VimoRAG improves FID from 0.501 to 0.131, Top1 R-Precision from 0.396 to 0.452, Top2 from 0.575 to 0.655, Top3 from 0.673 to 0.764, and MM Dist from 3.724 to 3.146, while diversity remains roughly unchanged at 9.475 versus 9.424 (Xu et al., 16 Aug 2025).

Ablations make the retriever’s contribution explicit. On HumanML3D validation, the full system (Gem+Mc) reaches FID 0.148. Replacing Gemini‑MVR with random video (Ran+Mc) yields FID 0.544; replacing it with InternVideo retrieval (Int+Mc) yields 0.205; removing McDPO (Gem) yields 0.260. The paper interprets these comparisons as showing that Gemini‑MVR materially improves the quality of the prior supplied to the motion generator, and that McDPO further improves how that prior is used (Xu et al., 16 Aug 2025).

6. Limitations, adjacent research, and significance

The paper identifies five main limitations of Gemini‑MVR. First, the action encoder depends on AlphaPose, so occlusion and extreme poses can degrade keypoint quality. Second, the representation is fundamentally 2D-centric and may lose depth or fine 3D kinematics. Third, retriever and generator are trained separately, so residual misalignment remains possible even with McDPO. Fourth, Gemini‑MVR processes up to 16 frames per video during training, which may miss longer temporal structures. Fifth, training the action-level retriever on HcVD with large batch sizes such as 2048 is computationally nontrivial, even if inference latency is modest relative to motion generation (Xu et al., 16 Aug 2025).

The future directions suggested in VimoRAG are correspondingly structural: larger-scale video corpora, integration of additional modalities such as 3D motion databases and images, tighter retriever–generator alignment objectives, and systematic evaluation of alternative LLM backbones for motion RAG (Xu et al., 16 Aug 2025). This suggests that Gemini‑MVR is best understood not as a finished universal retriever, but as a specialized motion-RAG module whose performance scales with both corpus size and alignment quality.

In broader context, Gemini‑MVR occupies a specific position among motion-centered retrieval systems. Earlier work on event retrieval emphasized viewpoint-invariant motion presence through motion barcodes, a binary per-pixel representation of motion existence over time (Ben-Artzi et al., 2014). More recent retrieval-augmented generation systems use retrieved videos as motion priors for video synthesis: RAGME conditions text-to-video diffusion models on retrieved videos through temporal cross-attention (Peruzzo et al., 9 Apr 2025), while MotionRAG introduces Context-Aware Motion Adaptation and motion injection adapters for image-to-video generation (Zhu et al., 30 Sep 2025). Native multimodal embedding models such as Gemini Embedding 2 place text, image, audio, and video into a unified retrieval space and report strong text-to-video retrieval performance on VATEX, MSR-VTT, and YouCook2 (Shanbhogue et al., 26 May 2026). Efficient Motion-Aware Video MLLMs, by contrast, use compressed-domain motion vectors and GOP encoders to improve motion sensitivity in video understanding (Zhao et al., 17 Mar 2025). Against this background, Gemini‑MVR is distinctive in being a human-motion-specialized retriever whose output is used not to retrieve videos as an end in itself, but to augment a motion LLM that generates 3D motion sequences (Xu et al., 16 Aug 2025).

Taken together, these properties define Gemini‑MVR as a motion-centric retrieval mechanism for retrieval-augmented 3D motion generation: dual-channel in architecture, keypoint-aware in routing, trained with contrastive alignment, and coupled to a generator that is explicitly optimized to exploit or disregard retrieved video priors depending on their utility.