--- title: 'Gemini‑MVR: 3D Motion Video Retriever' url: https://www.emergentmind.com/topics/gemini-motion-video-retriever type: topic --- # Gemini‑MVR: 3D Motion Video Retriever The **Gemini Motion Video Retriever** (Gemini‑MVR) is the retrieval component of VimoRAG, a video-based retrieval-augmented framework for 3D motion generation with motion large language models. In VimoRAG, Gemini‑MVR converts a text motion description into a relevant **in-the-wild video** from the Human-centric Video Database (HcVD), so that the downstream motion LLM can condition on retrieved 2D human motion signals rather than relying on text alone. Its stated purpose is to alleviate severe out-of-domain and out-of-vocabulary failure modes in motion LLMs trained on small text–motion corpora such as HumanML3D, which contains only about 14k text–motion pairs [2508.12081]. ## 1. Functional role and problem setting Within VimoRAG, Gemini‑MVR is the front-end retriever in a two-stage pipeline. Given a motion description \(t\), it searches HcVD and returns the best matching video \(v^\*\), specifically the rank-1 retrieval used in the paper. The motion LLM, implemented with Phi‑3‑3.8B plus a visual adapter, then consumes both the text prompt and the retrieved video, encoded into segment-wise embeddings \(E^v\), and generates discrete motion tokens \(y=\{y_1,\dots,y_{n_y}\}\), which are finally decoded by a VQ‑VAE into a 3D motion sequence [2508.12081]. The retriever is motivated by a specific limitation of existing motion LLMs. Systems such as MotionGPT are trained on relatively small text–motion datasets and therefore encounter two failure classes: **out-of-domain** prompts describing actions or styles absent from training, and **out-of-vocabulary** prompts involving unseen words or long composite descriptions. VimoRAG treats large-scale video corpora as a source of motion priors, but it also argues that video-based motion retrieval is nontrivial because generic video foundation models excel at object and scene recognition yet struggle with fine-grained human poses and behavior-only queries, particularly when object or environment cues are weak [2508.12081]. A common misconception is that generic text-to-video retrieval is sufficient for motion augmentation. The VimoRAG formulation rejects that premise explicitly. It frames motion retrieval as requiring human-centric matching of descriptions such as “running sideways while looking back” or “seated chopping wood, lift and strike,” and it further emphasizes robustness to partially mismatched retrievals because retrieval errors can propagate into the generator unless the model learns to discount them [2508.12081]. ## 2. Dual-retriever architecture and motion representations Gemini‑MVR is built as a **CLIP-style dual-encoder** retrieval system with two independent channels: an **object-level retriever** and an **action-level retriever**. The object-level branch uses InternVideo-derived text and video encoders, denoted \(\theta_{\mathcal G}\) and \(\theta_{\mathcal O}\), to represent generic semantic content. The action-level branch uses a text encoder \(\theta_{\mathcal P}\) together with a keypoint-based video encoder \(\theta_{\mathcal A}\), initialized from AlphaPose for 2D keypoint detection, MotionBERT as the keypoint encoder backbone, and a temporal Transformer [2508.12081]. The action branch represents a video \(v\) with \(m\) frames as a sequence of 2D body keypoints, \[ K(v)=\{K_1,K_2,\dots,K_m\}, \qquad K_i\in\mathbb{R}^{J\times 2}. \] Each frame is encoded by MotionBERT and projected into a 768-dimensional feature, \[ h_i=f_{\text{MBERT}}(K_i)\in\mathbb{R}^{8704}, \qquad a_i=W_{\text{proj}}h_i\in\mathbb{R}^{768}. \] After adding learnable positional embeddings \(p_i\), the sequence \(\tilde a_i=a_i+p_i\) is passed through a temporal Transformer with 4 layers, 12 heads, and width 768, yielding framewise outputs \(\hat a_i\). Mean pooling then forms the video-level action embedding, \[ \mathbf a=\frac{1}{m}\sum_{i=1}^{m}\hat a_i\in\mathbb{R}^{768}. \] This \(\mathbf a\) is the action representation used for similarity computation [2508.12081]. Text is encoded twice. The **predicate-semantic** representation is \[ \mathbf p=\theta_{\mathcal P}(t)\in\mathbb{R}^{d}, \] and the **argument-semantic** representation is \[ \mathbf g=\theta_{\mathcal G}(t)\in\mathbb{R}^{d}. \] On the video side, the object-level embedding is \[ \mathbf o=\theta_{\mathcal O}(v)\in\mathbb{R}^{d}. \] The paper makes an important qualification: despite the names “predicate” and “argument,” these text encoders do not hard-code predicate/argument parsing. Their specialization emerges implicitly through contrastive alignment with different video features [2508.12081]. This architecture is designed to separate human action cues from object cues. When a query is dominated by verbs, body parts, and temporal evolution, the action branch is intended to dominate; when object and environment semantics are informative, the object branch can contribute more strongly. The resulting system is therefore neither a pure keypoint retriever nor a pure foundation-model retriever. ## 3. Contrastive learning and keypoints-aware similarity routing The action-level retriever is trained with CLIP-style contrastive learning over text–video pairs. For a batch of size \(B\), with action-text embeddings \(\mathbf p_i\) and action-video embeddings \(\mathbf a_i\), cosine similarity is used: \[ s(\mathbf p,\mathbf a)=\frac{\mathbf p^\top\mathbf a}{\|\mathbf p\|\,\|\mathbf a\|}. \] The training objective is symmetric, consisting of text-to-action and action-to-text contrastive losses: \[ \mathcal L_{\text{action}}=\mathcal L_{p2a}+\mathcal L_{a2p}. \] The object-level retriever is trained analogously with \(\mathbf g_i\) and \(\mathbf o_i\), yielding \[ \mathcal L_{\text{object}}=\mathcal L_{g2o}+\mathcal L_{o2g}. \] These two branches are trained independently in the first training stage [2508.12081]. The distinctive component is the **action-aware similarity integrator** \(\mathcal I\), described as a lightweight integrator and also characterized as a **keypoints-aware router**. It receives the action embedding \(\mathbf a\) and outputs two scalars \((\mathcal I_0(\mathbf a),\mathcal I_1(\mathbf a))\), which softly gate the action-level and object-level similarities. The final retrieval score is \[ s(t,v)= \frac{\mathcal I_0(\mathbf a)\, s(\mathbf p,\mathbf a)}{\mathcal I_0(\mathbf a)+\mathcal I_1(\mathbf a)} + \frac{\mathcal I_1(\mathbf a)\, s(\mathbf g,\mathbf o)}{\mathcal I_0(\mathbf a)+\mathcal I_1(\mathbf a)}. \] The integrator is itself trained with a symmetric CLIP-style text–video contrastive loss, \[ \mathcal L_{\text{integ}}=\mathcal L_{t2v}+\mathcal L_{v2t}. \] At inference, Gemini‑MVR computes \(s(t,v)\) for all videos in HcVD, ranks them, and selects the top-1 video \(v^\*\) [2508.12081]. This design directly addresses noisy retrieval. Rather than fixing a static weighting between action and object similarity, Gemini‑MVR conditions the fusion on the action embedding itself. The paper’s claim is that videos with more prominent human motion signals receive higher action weight, which improves retrieval quality and reduces downstream error propagation [2508.12081]. ## 4. Conditioning the motion LLM and interaction with McDPO Once Gemini‑MVR retrieves \(v^\*\), the downstream generator treats the video as a prior rather than as an obligatory template. The retrieved video is encoded by InternVideo2 into segment-wise embeddings \[ E^v=\{E^v_1,\dots,E^v_k\}, \] which are combined with the system prompt \(\mathcal P\), instruction template \(\mathcal T\), and text input \(x=\{x_1,\dots,x_{n_x}\}\) to form \[ E^f=[\mathrm{emb}(\mathcal P),\ \mathcal T(\mathrm{emb}(x),E^v)]. \] The instruction format used in the paper is: “Generate a sequence of motion tokens matching the following human motion description. You can use the video as a reference. Video information: \{Retrieved Video v\} Motion description: \{Input Text x\}” [2508.12081]. During supervised fine-tuning, the target 3D motion is quantized by a VQ‑VAE into discrete tokens \(y=\{y_1,\dots,y_{n_y}\}\), and the model is trained with cross-entropy, \[ \mathcal L_{\text{sft}}=-\sum_n \log p_\theta(y_n\mid y_{