ReAlign: Bilingual Text-to-Motion Sampling

• ReAlign is a sampling-time technique that uses a step-aware reward model to adjust reverse diffusion in bilingual text-to-motion generation.
• It integrates a unified backbone (BiMD) and a bilingual dataset (BiHumanML3D) to align English and Chinese text with corresponding 3D human motions.
• The method dynamically combines text-aligned and motion-aligned rewards during denoising to significantly improve metrics like FID and R-Precision.

ReAlign, sometimes confused with “ResAlign,” is the alignment mechanism introduced in “ReAlign: Bilingual Text-to-Motion Generation via Step-Aware Reward-Guided Alignment” (Weng et al., 8 May 2025). It is a plug-and-play reward-guided sampling method for diffusion models in bilingual text-to-motion generation, where the input is an English or Chinese text description and the output is a 3D human motion sequence. In the paper’s system, ReAlign is not the backbone generator itself: it operates on top of a bilingual motion diffusion model called BiMD, and that backbone in turn is enabled by a bilingual dataset called BiHumanML3D. The method is motivated by two coupled difficulties: the prior absence of a public bilingual text-motion dataset, and a distribution misalignment between text conditioning and motion generation in diffusion models, which can yield semantically inconsistent or low-quality motions (Weng et al., 8 May 2025).

1. Task formulation and system context

The paper studies bilingual text-to-motion generation, defined as generating 3D human motion from text in either English or Chinese. It argues that the bilingual version of the task is harder than the monolingual case for two reasons. First, there was previously no public bilingual text-motion dataset. Second, diffusion-based motion generators often condition on encoders such as CLIP that were aligned to images rather than motion, which creates a distribution misalignment: the model samples from a high-probability motion distribution learned from data, but that distribution is not necessarily the one that best matches the text prompt.

The system is organized into three components. BiHumanML3D provides bilingual supervision. BiMD is a unified bilingual diffusion backbone conditioned on cross-lingually aligned text representations. ReAlign is attached at inference time and modifies the reverse denoising trajectory by adding a reward gradient. A useful correction to common usage is therefore that ReAlign is fundamentally a sampling-time method, not a retraining or fine-tuning procedure for the generator itself. However, it does require a separately trained reward model.

A concise way to summarize the pipeline is: bilingual text input, then bilingual text encoding with cross-lingual alignment, then diffusion-based motion generation via BiMD, then reverse denoising steps modified by ReAlign’s reward gradient, and finally the motion output. This suggests that ReAlign is best understood as an alignment layer over an existing generator rather than as a replacement for the generator.

2. Bilingual data and the BiMD backbone

The dataset introduced for this setting is BiHumanML3D, which extends HumanML3D into a bilingual benchmark by keeping the original motions and adding Chinese descriptions aligned with the English ones. The paper states that BiHumanML3D contains 13,312 bilingual motions. In the supplementary material, it also reiterates that HumanML3D has 14,616 motion sequences and 44,970 annotations, and that BiHumanML3D preserves the original motions while extending the text side to English and Chinese.

The Chinese annotations are produced through a three-stage LLM-assisted pipeline: initial translation with DeepSeek, refinement with Qwen, and final validation and correction by DeepSeek plus human annotators. The paper states that it filters out overly short and overly long motion clips following prior work, but the exact train/validation/test split is not explicitly stated in the provided text.

BiMD is the bilingual motion diffusion backbone trained on this dataset. Its central design choice is to use one unified diffusion model rather than separate English and Chinese generators. Cross-lingual conditioning is obtained through a teacher-student alignment stage in which a student encoder $E^s_\phi$, based on XLM, is aligned to a teacher encoder $E^t_\Phi$, based on OpenCLIP. The paper describes this as a symmetric KL-style alignment between English teacher features and Chinese student features, making the two languages interchangeable as conditioning signals.

During BiMD training, each motion sample is randomly paired with either its English or Chinese description, so the model learns a language-agnostic conditional motion distribution. The diffusion loss is

$\mathcal{L}_{\text{BiMD} (\theta) = \mathbb{E}_{\epsilon,t,c} \left[\left\| \epsilon - \epsilon_\theta \left(\mathbf{x}_t,t, E_{\phi}^{s}\left(c_{s}\right)\right) \right\|^{2}_{2}\right],$

where $c_s$ is the selected English or Chinese caption and $E^s_\phi(c_s)$ is its aligned text feature.

Component	Function	Stated role
BiHumanML3D	Dataset	Bilingual supervision and benchmark
BiMD	Backbone	Unified bilingual motion diffusion model
ReAlign	Inference-time alignment	Reward-guided sampling during denoising

3. Reward-guided alignment as modified reverse diffusion

The core conceptual claim of ReAlign is that one can improve semantic faithfulness by augmenting the diffusion model’s sampling distribution with a learned reward distribution (Weng et al., 8 May 2025). The paper writes the continuous-time forward and reverse processes as

$$\mathrm{d}\mathbf{x} = \mathbf{f}(\mathbf{x},t) \mathrm{d}t + g(t) \mathrm{d}\mathbf{w},$$

and

$$\mathrm{d}\mathbf{x} = [\mathbf{f}(\mathbf{x},t)- g(t)^2\nabla \log p_t(\mathbf{x})] \mathrm{d}t + g(t) \mathrm{d}\mathbf{w}.$$

ReAlign then defines an “ideal” sampling distribution

$$p_t^{I}(\mathbf{x}|c) = p_t(\mathbf{x}|c) p_t^{r}(\mathbf{x}|c)/ Z(c),$$

with

$$Z(c) = \int p_t(\mathbf{x}|c) p_t^{r}(\mathbf{x}|c)\mathrm{d}\mathbf{x}.$$

This yields the modified reverse SDE

$$\mathbf{dx} = \Big[\mathbf{f}(\mathbf{x},t)- g(t)^2 \nabla \big( \log p_t(\mathbf{x}|c)+\log p_t^{r}(\mathbf{x}|c) \big)\Big]\mathrm{d}t + g(t) \mathrm{d}\mathbf{w}.$$

The significance of this construction is that the reward is not appended as a post hoc ranking signal. Instead, its gradient is injected directly into reverse diffusion. In the paper’s framing, the diffusion score still guides sampling toward high-density motion regions, but the reward term steers the trajectory toward motions that are both more text-aligned and more realistic. This suggests that ReAlign is closer to score modification than to classical reranking.

A frequent misconception is that this sort of alignment requires retraining the generator for every new reward. The paper’s design rejects that premise: the generator can remain fixed, while a separate reward model defines $E^t_\Phi$0 and contributes its gradient during sampling.

4. Step-aware reward model and dual reward structure

The main novelty of ReAlign lies in how it estimates the reward on noisy intermediate motions, not only on clean final outputs. The paper therefore introduces a step-aware reward model. Operationally, this means appending a timestep token $E^t_\Phi$1 to the noisy motion sequence $E^t_\Phi$2, yielding $E^t_\Phi$3, and processing that sequence jointly with text using a transformer encoder.

The reward model is trained with

$E^t_\Phi$4

Here, $E^t_\Phi$5 is a contrastive text-motion alignment loss, and $E^t_\Phi$6 is a representation or reconstruction-style loss inherited from prior motion-language representation work. The contrastive term uses matched text-motion pairs as positives and other examples in the batch as negatives, filtered by a sentence similarity threshold.

The total reward has two modules. The first is the text-aligned module

$E^t_\Phi$7

where $E^t_\Phi$8 and $E^t_\Phi$9 are motion and text embeddings. This is the semantic consistency term. The second is the motion-aligned module, which retrieves a reference motion from the training set using the text-aligned reward: $\mathcal{L}_{\text{BiMD} (\theta) = \mathbb{E}_{\epsilon,t,c} \left[\left\| \epsilon - \epsilon_\theta \left(\mathbf{x}_t,t, E_{\phi}^{s}\left(c_{s}\right)\right) \right\|^{2}_{2}\right],$0 The corresponding realism term is

$\mathcal{L}_{\text{BiMD} (\theta) = \mathbb{E}_{\epsilon,t,c} \left[\left\| \epsilon - \epsilon_\theta \left(\mathbf{x}_t,t, E_{\phi}^{s}\left(c_{s}\right)\right) \right\|^{2}_{2}\right],$1

The full reward is then

$\mathcal{L}_{\text{BiMD} (\theta) = \mathbb{E}_{\epsilon,t,c} \left[\left\| \epsilon - \epsilon_\theta \left(\mathbf{x}_t,t, E_{\phi}^{s}\left(c_{s}\right)\right) \right\|^{2}_{2}\right],$2

which induces the reward distribution

$\mathcal{L}_{\text{BiMD} (\theta) = \mathbb{E}_{\epsilon,t,c} \left[\left\| \epsilon - \epsilon_\theta \left(\mathbf{x}_t,t, E_{\phi}^{s}\left(c_{s}\right)\right) \right\|^{2}_{2}\right],$3

This decomposition is important for interpreting the method. The paper’s ablations state that the text-aligned reward is the primary driver of semantic improvements, while the motion-aligned reward especially helps realism, and the step-aware design makes reward estimation reliable across diffusion timesteps. The reward model itself is trained using existing text-motion pairs only; the paper explicitly states that it does not require extra human preference annotations or new labels.

Architecturally, the reward model is based on SkipTransformer, with transformer encoder and decoder each having 9 layers, 4 heads, and latent dimension 256. Training uses maximum timestep 1000, noisy motion probability 0.5, AdamW, learning rate $\mathcal{L}_{\text{BiMD} (\theta) = \mathbb{E}_{\epsilon,t,c} \left[\left\| \epsilon - \epsilon_\theta \left(\mathbf{x}_t,t, E_{\phi}^{s}\left(c_{s}\right)\right) \right\|^{2}_{2}\right],$4, and batch size 128.

5. Sampling rule, plug-and-play use, and empirical behavior

Once the reward is available, ReAlign modifies each denoising step by adding the reward gradient. The paper derives a DDPM-style discrete update, notes that the theoretically weighted reward term can destabilize early noisy steps, and therefore uses a practical update in which the reward gradient is added directly to each denoising update. In procedural terms, sampling first retrieves the reference motion $\mathcal{L}_{\text{BiMD} (\theta) = \mathbb{E}_{\epsilon,t,c} \left[\left\| \epsilon - \epsilon_\theta \left(\mathbf{x}_t,t, E_{\phi}^{s}\left(c_{s}\right)\right) \right\|^{2}_{2}\right],$5, then computes reward values at every timestep, then updates the sample with the modified rule inside the denoising loop.

This inference-time character explains why ReAlign is described as plug-and-play. The method is demonstrated on the bilingual BiMD backbone, but it is also applied to other generators. On HumanML3D, BiMD alone achieves $\mathcal{L}_{\text{BiMD} (\theta) = \mathbb{E}_{\epsilon,t,c} \left[\left\| \epsilon - \epsilon_\theta \left(\mathbf{x}_t,t, E_{\phi}^{s}\left(c_{s}\right)\right) \right\|^{2}_{2}\right],$6-Precision Top-1/2/3 of 0.499/0.691/0.789, FID 0.397, MM Dist 3.105, and Diversity 9.635; adding ReAlign improves these to 0.566/0.759/0.847, FID 0.178, MM Dist 2.714, and Diversity 9.573. The paper highlights this as a 55.2% FID improvement over BiMD without additional training (Weng et al., 8 May 2025).

The plug-and-play claim is supported further by other backbones. For MLD++, ReAlign improves Top-3 from 0.829 to 0.852, FID from 0.073 to 0.055, and MM Dist from 2.810 to 2.648. For MDM on KIT-ML, it improves Top-3 from 0.731 to 0.784, FID from 0.497 to 0.276, and MM Dist from 3.096 to 2.775.

The evaluation metrics are standard text-to-motion metrics. R-Precision measures text-motion retrieval correctness; FID measures the distance between generated and real motion feature distributions; MM Dist measures the distance between a generated motion and its text condition in evaluation space; Diversity measures spread among generated motions. The bilingual results further indicate that BiMD with cross-lingual alignment outperforms adapted monolingual or language-specific baselines for both Chinese and English, which matters because ReAlign assumes a competent conditional backbone.

A second misconception is that the motion-aligned module alone drives most of the gain. The ablations do not support that view. Starting from BiMD without ReAlign, adding only text-to-motion reward gives a large jump in alignment and FID; adding only motion-to-motion reward helps less on its own; combining both helps more; and adding SA step-aware training yields the best overall result.

6. Interpretation, limitations, and relation to adjacent methods

ReAlign occupies a specific place in the text-to-motion literature. It is neither a new bilingual dataset alone nor merely a bilingual diffusion model. Its distinct contribution is a step-aware reward-guided alignment mechanism for diffusion-based text-to-motion generation that acts during sampling. In direct terms, it estimates rewards on noisy intermediate motions, combines semantic consistency with realism through retrieved reference motions, turns that combined reward into a reward distribution, and injects the resulting gradient into reverse diffusion.

Several clarifications follow from that formulation. ReAlign is not an RLHF-style method. The paper explicitly states that it does not require extra human preference annotations or new labels. It is also not a generator retraining method, although it does require training a separate reward model beforehand. Finally, despite being developed for bilingual generation, it is not tied only to BiMD; the reported results on MLD++ and MDM are intended to show broader plug-and-play applicability.

The limitations are also clearly delimited. The authors note that the broader motion generation stack remains constrained by limited available motion data, which can limit the quality of pretrained motion representations such as VAE-like latent spaces and therefore limit generalization. ReAlign also adds inference overhead, because it requires reward computation and gradient guidance at every denoising step, plus retrieval of a reference motion $\mathcal{L}_{\text{BiMD} (\theta) = \mathbb{E}_{\epsilon,t,c} \left[\left\| \epsilon - \epsilon_\theta \left(\mathbf{x}_t,t, E_{\phi}^{s}\left(c_{s}\right)\right) \right\|^{2}_{2}\right],$7. The exact runtime overhead is not quantified in the provided text. The paper also does not report a formal human user study in the provided text.

A final interpretive point concerns the name itself. In this context, “ResAlign” is best treated as a mistaken rendering of ReAlign. The method introduced in (Weng et al., 8 May 2025) is “ReAlign,” and within that paper the term denotes a reward-guided, step-aware sampling-time alignment procedure for bilingual text-to-motion diffusion models rather than a residual-alignment method, a reference-video alignment method, or a multilingual encoder realignment method.