---
title: 'ReAlign: Bilingual Text-to-Motion Sampling'
url: https://www.emergentmind.com/topics/resalign
type: topic
---

# ReAlign: Bilingual Text-to-Motion Sampling

ReAlign, sometimes confused with “ResAlign,” is the alignment mechanism introduced in “ReAlign: Bilingual Text-to-Motion Generation via Step-Aware Reward-Guided Alignment” [2505.04974]. It is a **plug-and-play reward-guided sampling method for diffusion models** in **bilingual text-to-motion generation**, where the input is an English or Chinese text description and the output is a 3D human motion sequence. In the paper’s system, ReAlign is not the backbone generator itself: it operates on top of a bilingual motion diffusion model called **BiMD**, and that backbone in turn is enabled by a bilingual dataset called **BiHumanML3D**. The method is motivated by two coupled difficulties: the prior absence of a public bilingual text-motion dataset, and a **distribution misalignment** between text conditioning and motion generation in diffusion models, which can yield semantically inconsistent or low-quality motions [2505.04974].

## 1. Task formulation and system context

The paper studies **bilingual text-to-motion generation**, defined as generating 3D human motion from text in either **English or Chinese**. It argues that the bilingual version of the task is harder than the monolingual case for two reasons. First, there was previously **no public bilingual text-motion dataset**. Second, diffusion-based motion generators often condition on encoders such as CLIP that were aligned to images rather than motion, which creates a **distribution misalignment**: the model samples from a high-probability motion distribution learned from data, but that distribution is not necessarily the one that best matches the text prompt.

The system is organized into three components. **BiHumanML3D** provides bilingual supervision. **BiMD** is a unified bilingual diffusion backbone conditioned on cross-lingually aligned text representations. **ReAlign** is attached at inference time and modifies the reverse denoising trajectory by adding a reward gradient. A useful correction to common usage is therefore that ReAlign is fundamentally a **sampling-time method**, not a retraining or fine-tuning procedure for the generator itself. However, it does require a separately trained reward model.

A concise way to summarize the pipeline is: bilingual text input, then bilingual text encoding with cross-lingual alignment, then diffusion-based motion generation via BiMD, then reverse denoising steps modified by ReAlign’s reward gradient, and finally the motion output. This suggests that ReAlign is best understood as an alignment layer over an existing generator rather than as a replacement for the generator.

## 2. Bilingual data and the BiMD backbone

The dataset introduced for this setting is **BiHumanML3D**, which extends **HumanML3D** into a bilingual benchmark by keeping the original motions and adding Chinese descriptions aligned with the English ones. The paper states that BiHumanML3D contains **13,312 bilingual motions**. In the supplementary material, it also reiterates that HumanML3D has **14,616 motion sequences** and **44,970 annotations**, and that BiHumanML3D preserves the original motions while extending the text side to **English and Chinese**.

The Chinese annotations are produced through a three-stage LLM-assisted pipeline: initial translation with **DeepSeek**, refinement with **Qwen**, and final validation and correction by **DeepSeek plus human annotators**. The paper states that it filters out overly short and overly long motion clips following prior work, but the exact train/validation/test split is **not explicitly stated** in the provided text.

BiMD is the bilingual motion diffusion backbone trained on this dataset. Its central design choice is to use **one unified diffusion model** rather than separate English and Chinese generators. Cross-lingual conditioning is obtained through a teacher-student alignment stage in which a student encoder \(E^s_\phi\), based on **XLM**, is aligned to a teacher encoder \(E^t_\Phi\), based on **OpenCLIP**. The paper describes this as a symmetric KL-style alignment between English teacher features and Chinese student features, making the two languages interchangeable as conditioning signals.

During BiMD training, each motion sample is randomly paired with either its English or Chinese description, so the model learns a language-agnostic conditional motion distribution. The diffusion loss is
\[
\mathcal{L}_{\text{BiMD} (\theta) = \mathbb{E}_{\epsilon,t,c} \left[\left\| \epsilon - \epsilon_\theta \left(\mathbf{x}_t,t, E_{\phi}^{s}\left(c_{s}\right)\right) \right\|^{2}_{2}\right],
\]
where \(c_s\) is the selected English or Chinese caption and \(E^s_\phi(c_s)\) is its aligned text feature.

| Component | Function | Stated role |
|---|---|---|
| **BiHumanML3D** | Dataset | Bilingual supervision and benchmark |
| **BiMD** | Backbone | Unified bilingual motion diffusion model |
| **ReAlign** | Inference-time alignment | Reward-guided sampling during denoising |

## 3. Reward-guided alignment as modified reverse diffusion

The core conceptual claim of ReAlign is that one can improve semantic faithfulness by augmenting the diffusion model’s sampling distribution with a learned reward distribution [2505.04974]. The paper writes the continuous-time forward and reverse processes as
\[
\mathrm{d}\mathbf{x} = \mathbf{f}(\mathbf{x},t) \mathrm{d}t + g(t) \mathrm{d}\mathbf{w},
\]
and
\[
\mathrm{d}\mathbf{x} = [\mathbf{f}(\mathbf{x},t)- g(t)^2\nabla \log p_t(\mathbf{x})] \mathrm{d}t + g(t) \mathrm{d}\mathbf{w}.
\]

ReAlign then defines an “ideal” sampling distribution
\[
p_t^{I}(\mathbf{x}|c) = p_t(\mathbf{x}|c) p_t^{r}(\mathbf{x}|c)/ Z(c),
\]
with
\[
Z(c) = \int p_t(\mathbf{x}|c) p_t^{r}(\mathbf{x}|c)\mathrm{d}\mathbf{x}.
\]
This yields the modified reverse SDE
\[
\mathbf{dx} = \Big[\mathbf{f}(\mathbf{x},t)- g(t)^2 \nabla \big( \log p_t(\mathbf{x}|c)+\log p_t^{r}(\mathbf{x}|c) \big)\Big]\mathrm{d}t + g(t) \mathrm{d}\mathbf{w}.
\]

The significance of this construction is that the reward is not appended as a post hoc ranking signal. Instead, its gradient is injected directly into reverse diffusion. In the paper’s framing, the diffusion score still guides sampling toward high-density motion regions, but the reward term steers the trajectory toward motions that are both more text-aligned and more realistic. This suggests that ReAlign is closer to score modification than to classical reranking.

A frequent misconception is that this sort of alignment requires retraining the generator for every new reward. The paper’s design rejects that premise: the generator can remain fixed, while a separate reward model defines \(p_t^r(\mathbf{x}\mid c)\) and contributes its gradient during sampling.

## 4. Step-aware reward model and dual reward structure

The main novelty of ReAlign lies in how it estimates the reward on **noisy intermediate motions**, not only on clean final outputs. The paper therefore introduces a **step-aware reward model**. Operationally, this means appending a timestep token \( [e_t] \) to the noisy motion sequence \([x_t^1,\dots,x_t^N]\), yielding \([e_t, x_t^1, x_t^2, \dots, x_t^N]\), and processing that sequence jointly with text using a transformer encoder.

The reward model is trained with
\[
\mathcal{L}_{RM}(\varphi; \mathbf{x}_t, c) = \mathcal{L}_{C}(\varphi; \mathbf{x}_t, c) + \mathcal{L}_{R}(\varphi; \mathbf{x}_t, c).
\]
Here, \(\mathcal{L}_C\) is a contrastive text-motion alignment loss, and \(\mathcal{L}_R\) is a representation or reconstruction-style loss inherited from prior motion-language representation work. The contrastive term uses matched text-motion pairs as positives and other examples in the batch as negatives, filtered by a sentence similarity threshold.

The total reward has two modules. The first is the **text-aligned module**
\[
R_{\varphi} (\mathbf{x}, c) = \cos(\mathbf{z_x}, \mathbf{z}_c),
\]
where \(\mathbf{z_x}\) and \(\mathbf{z}_c\) are motion and text embeddings. This is the semantic consistency term. The second is the **motion-aligned module**, which retrieves a reference motion from the training set using the text-aligned reward:
\[
\mathbf{x}^c = \arg \max_{\mathbf{x}\in\mathcal{D}_{tr}} R_\varphi(\mathbf{x}, c).
\]
The corresponding realism term is
\[
R_m\left(\mathbf{x}_t, c\right) = \cos(\mathbf{z_x}, \mathbf{z}_{x^c}).
\]

The full reward is then
\[
R \left( \mathbf{x}_t, c\right) = \mu R_\varphi(\mathbf{x}_t, c) + \eta R_m\left(\mathbf{x}_t, c\right),
\]
which induces the reward distribution
\[
p^r_t(\mathbf{x}_t|c) = \exp \left(R \left(\mathbf{x}_t, c\right) \right) / {Z^r(c)}.
\]

This decomposition is important for interpreting the method. The paper’s ablations state that the **text-aligned reward is the primary driver of semantic improvements**, while the **motion-aligned reward especially helps realism**, and the **step-aware design** makes reward estimation reliable across diffusion timesteps. The reward model itself is trained using existing text-motion pairs only; the paper explicitly states that it does **not** require extra human preference annotations or new labels.

Architecturally, the reward model is based on **SkipTransformer**, with transformer encoder and decoder each having **9 layers**, **4 heads**, and latent dimension **256**. Training uses **maximum timestep 1000**, **noisy motion probability 0.5**, **AdamW**, learning rate **\(10^{-4}\)**, and batch size **128**.

## 5. Sampling rule, plug-and-play use, and empirical behavior

Once the reward is available, ReAlign modifies each denoising step by adding the reward gradient. The paper derives a DDPM-style discrete update, notes that the theoretically weighted reward term can destabilize early noisy steps, and therefore uses a practical update in which the reward gradient is added directly to each denoising update. In procedural terms, sampling first retrieves the reference motion \(\mathbf{x}^c\), then computes reward values at every timestep, then updates the sample with the modified rule inside the denoising loop.

This inference-time character explains why ReAlign is described as **plug-and-play**. The method is demonstrated on the bilingual **BiMD** backbone, but it is also applied to other generators. On **HumanML3D**, BiMD alone achieves \(R\)-Precision Top-1/2/3 of **0.499/0.691/0.789**, FID **0.397**, MM Dist **3.105**, and Diversity **9.635**; adding ReAlign improves these to **0.566/0.759/0.847**, FID **0.178**, MM Dist **2.714**, and Diversity **9.573**. The paper highlights this as a **55.2% FID improvement** over BiMD without additional training [2505.04974].

The plug-and-play claim is supported further by other backbones. For **MLD++**, ReAlign improves Top-3 from **0.829** to **0.852**, FID from **0.073** to **0.055**, and MM Dist from **2.810** to **2.648**. For **MDM** on **KIT-ML**, it improves Top-3 from **0.731** to **0.784**, FID from **0.497** to **0.276**, and MM Dist from **3.096** to **2.775**.

The evaluation metrics are standard text-to-motion metrics. **R-Precision** measures text-motion retrieval correctness; **FID** measures the distance between generated and real motion feature distributions; **MM Dist** measures the distance between a generated motion and its text condition in evaluation space; **Diversity** measures spread among generated motions. The bilingual results further indicate that BiMD with cross-lingual alignment outperforms adapted monolingual or language-specific baselines for both Chinese and English, which matters because ReAlign assumes a competent conditional backbone.

A second misconception is that the motion-aligned module alone drives most of the gain. The ablations do not support that view. Starting from BiMD without ReAlign, adding only text-to-motion reward gives a large jump in alignment and FID; adding only motion-to-motion reward helps less on its own; combining both helps more; and adding **SA** step-aware training yields the best overall result.

## 6. Interpretation, limitations, and relation to adjacent methods

ReAlign occupies a specific place in the text-to-motion literature. It is neither a new bilingual dataset alone nor merely a bilingual diffusion model. Its distinct contribution is a **step-aware reward-guided alignment mechanism for diffusion-based text-to-motion generation** that acts during sampling. In direct terms, it estimates rewards on noisy intermediate motions, combines semantic consistency with realism through retrieved reference motions, turns that combined reward into a reward distribution, and injects the resulting gradient into reverse diffusion.

Several clarifications follow from that formulation. ReAlign is **not** an RLHF-style method. The paper explicitly states that it does **not** require extra human preference annotations or new labels. It is also **not** a generator retraining method, although it does require training a separate reward model beforehand. Finally, despite being developed for bilingual generation, it is **not tied only to BiMD**; the reported results on MLD++ and MDM are intended to show broader plug-and-play applicability.

The limitations are also clearly delimited. The authors note that the broader motion generation stack remains constrained by limited available motion data, which can limit the quality of pretrained motion representations such as VAE-like latent spaces and therefore limit generalization. ReAlign also adds inference overhead, because it requires reward computation and gradient guidance at every denoising step, plus retrieval of a reference motion \(\mathbf{x}^c\). The exact runtime overhead is **not quantified** in the provided text. The paper also does **not** report a formal human user study in the provided text.

A final interpretive point concerns the name itself. In this context, “ResAlign” is best treated as a mistaken rendering of **ReAlign**. The method introduced in [2505.04974] is “ReAlign,” and within that paper the term denotes a reward-guided, step-aware sampling-time alignment procedure for bilingual text-to-motion diffusion models rather than a residual-alignment method, a reference-video alignment method, or a multilingual encoder realignment method.

Source: https://www.emergentmind.com/topics/resalign