In-Context Model Predictive Generation: Open-Vocabulary Motion Synthesis from Language Models to Physics

Abstract: Synthesizing human motion from textual descriptions is essential for immersive digital applications, yet existing methods face a persistent trade-off between semantic fidelity and physical realism. LLM-based approaches can interpret diverse open-vocabulary instructions and compose high-level action plans, but they often generate motions that violate physical constraints. Physics-aware models improve realism through simulation or control, but they struggle with semantic complexity, fine-grained instructions, and novel concepts. To address this gap, we propose In-Context Model Predictive Generation (ICMPG), a framework that integrates language-model planning with inference-time physical feedback. ICMPG reformulates motion synthesis as a Model Predictive Control (MPC)-like process with two modules. The Context-Aware Motion Generation (CAMG) module uses an LLM as a planner to decompose textual commands and generate candidate motion sequences from motion tokens. The Model Predictive Generation (MPG) module evaluates these candidates through physical simulation and semantic alignment, estimates a composite reward, and selects the best sequence to guide subsequent generation steps. Unlike open-loop generation, this closed-loop refinement enables ICMPG to adapt motions to both the input semantics and the simulated physical environment without task-specific policy retraining. Extensive experiments across standard and zero-shot open-vocabulary settings show that ICMPG generalizes robustly to diverse commands and produces motions that are more physically plausible and semantically faithful than representative baselines on the evaluated benchmarks. The framework bridges semantic interpretation and physical simulation while remaining flexible enough to incorporate different LLM backbones, enabling more versatile and controllable text-driven motion synthesis.

• The paper introduces ICMPG, a novel framework that fuses LLM planning with physics simulation to achieve semantically faithful and physically feasible motion synthesis.
• It employs a two-stage process with CAMG for candidate generation and MPG for iterative refinement using a composite semantic-physical reward function.
• Extensive experiments demonstrate improved R-Precision, CLIP scores, and robustness across benchmarks, enabling scalable, zero-shot motion synthesis.

In-Context Model Predictive Generation: Unifying LLMs and Physics for Open-Vocabulary Motion Synthesis

Motivation and Problem Statement

Synthesizing human motion from natural language is foundational for virtual humans in a range of domains, including gaming, simulation, and robotics. The central challenge is reconciling two desirable but conflicting properties: semantic faithfulness (precision to input open-vocabulary text) and physical realism (adherence to laws of physics). Prior LLM-based systems interpret rich, diverse instructions but generate motions that are semantically plausible yet often physically infeasible. Conversely, physics-based controllers robustly enforce constraints but are limited to narrow skill sets and struggle with semantically nuanced or unseen instructions.

The paper introduces In-Context Model Predictive Generation (ICMPG), a hybrid framework that closes the semantic-physical gap by reformulating motion synthesis as an inference-time, model-predictive optimization problem. ICMPG employs LLMs for compositional semantic reasoning and leverages closed-loop physical simulation for selection and refinement, uniting the strengths of both approaches.

ICMPG Framework Overview

The ICMPG pipeline consists of two tightly integrated modules:

1. Context-Aware Motion Generation (CAMG): Leverages LLMs, structured via in-context prompting, to decompose text prompts and generate candidate motion sequences as discrete token sequences. The LLM planner first normalizes free-form instructions to atomic action sequences, aided by retrieved demonstrations via a learned SemAligner.
2. Model Predictive Generation (MPG): Evaluates candidate motions from CAMG using a composite reward function that balances semantic alignment (computed by text-mesh encoders) and physical plausibility (simulated by a pre-trained physics policy in Isaac Gym). The highest-reward sequence is selected and appended. The process iterates to refine and extend the motion.

   Figure 2: ICMPG framework pipeline showing offline preparation (VQ-VAE training, SemAligner setup, physics policy selection) and the online text-to-motion loop integrating LLM planning, demonstration retrieval, candidate generation and closed-loop model-predictive refinement.

The framework is explicitly inspired by Model Predictive Control (MPC):

• In classical MPC, a controller generates candidate control sequences, predicts state evolution via a dynamics model, and applies only the best initial action—repeating iteratively.
• In ICMPG, "control actions" correspond to selecting motion tokens; the LLM serves as a high-level model, while the MPG module refines and commits over a finite horizon using continuous feedback.

Figure 1: The analogy between classical MPC and ICMPG, highlighting the receding-horizon principle and feedback optimization at the level of motion tokens rather than low-level control signals.

Technical Contributions

Motion Representation and Planning

Human motion is quantized into atomic primitives using a VQ-VAE, allowing the LLM to plan in a space where temporal composition can express diverse and complex behaviors. The LLM planner, frozen during motion generation, parses text into normalized atomic sequences using prior demonstrations selected by an online learned SemAligner.

Semantic retrieval ensures robust in-context learning by focusing the LLM's context window on relevant examples and reducing demonstration misalignment. This is critical for out-of-distribution and long-horizon prompts.

Model Predictive Generation with Physical and Semantic Rewards

The MPG module evaluates motion candidates over a finite horizon based on:

• Physical Reward ($\mathcal{R}_{phy}$): Success and fidelity of simulation using a generic physics policy, ensuring generated motions are dynamically feasible without retraining for each task.
• Semantic Reward ($\mathcal{R}_{sem}$): CLIP- or ActionCLIP-based text-mesh alignment, computed on key SMPL-rendered frames for domain-invariant comparison.

A weighted sum (parameterized by $\beta$) balances these rewards, and only candidates above a semantic threshold are accepted, mitigating hallucination and failure cases.

Figure 5: Effect of the balancing parameter $\beta$ on semantic/physical trade-off. Increasing $\beta$ improves text alignment at the cost of higher physical error.

Iterative Refinement and LoRA Fine-Tuning

Unlike open-loop LLM-based generation, ICMPG’s receding-horizon optimization enables iterative error correction. An optional LoRA fine-tuning pathway, with a differentiable simulator model, accelerates training and reduces inference cost at some expense to flexibility.

Figure 7: Qualitative improvement in input-text alignment as ICMPG’s iterative MPG process progresses.

Flexibility and Modular Design

ICMPG is inherently modular and LLM-agnostic: the quality of downstream generation improves as more capable LLMs or semantic evaluators (e.g., ActionCLIP, Gemini Pro) are adopted, evidenced by ablation studies on backbone models. The LLM planner is particularly crucial for current models; its role may diminish as foundation model capabilities increase.

Experimental Results

Extensive experiments are conducted on HumanML3D, KIT-ML, BABEL (zero-shot open-vocabulary setting), and challenging recent benchmarks such as MotionMillion and PP-Motion.

• On open-vocabulary tasks, ICMPG (with LoRA) achieves the highest R-Precision and CLIP Score, surpassing strong baselines such as AnySkill, MotionCLIP, and AvatarCLIP in both semantic (MM-Hit@1) and physical (Mod-Cost, Phys-Err) metrics.
• Success rates and low physical error are maintained without retraining or policy overfit, evidencing effective generalization.

Figure 8: Comparison of ICMPG, MotionCLIP, and AnySkill in physical simulation on representative inputs. ICMPG produces motions that are both semantically aligned and robust in simulation.

Figure 3: Qualitative alignment with open-vocabulary descriptions underlining the superior semantic fidelity of ICMPG.

On long-sequence generation, ICMPG maintains global semantic coherence while avoiding physically implausible transitions, highlighting closed-loop optimization’s benefit over single-pass LLM or contrastive methods.

Figure 4: ICMPG-generated long-horizon motions demonstrate consistent semantic progression and physically plausible segment transitions.

Ablation Analyses

• Codebook size in VQ-VAE: $512\times 512$ achieves optimal tradeoff between diversity and semantic distinction; further increases yield diminishing returns or minor overfitting.
• LLM backbone: Generation quality is highly sensitive to the underlying LLM’s reasoning power, with LLaMA-3 and Gemini outperforming LLaMA-2 and Phi-4.
• Planner module: Success rate drops dramatically without it, especially on complex BABEL prompts; new-generation LLMs mitigate but do not eliminate the need.
• Reward function robustness: Performance degrades gracefully under semantic or demonstration noise; ActionCLIP outperforms vanilla CLIP for reward computation.

  Figure 10: t-SNE visualization of codebook utilization shows substantial overlap between in- and out-of-distribution tokens, supporting compositional generalization via learned atomic motion primitives.

Theoretical Implications and Future Perspectives

ICMPG demonstrates that combining high-level planning from LLMs with physical simulation feedback at inference enables scalable, zero-shot motion synthesis, generalizing to instructions and kinematic regimes outside the training set. Its closed-loop, modular architecture provides a principled approach for integrating advances in representation, evaluation, and language understanding.

Practically, this approach enables flexible adaptation to novel tasks and instruction sets without retraining physics skills or end-to-end policies, a longstanding bottleneck in interactive agents and animation. The modular, plug-and-play design paves the way for rapid incorporation of advances in foundation models and multimodal evaluators.

Looking forward, several avenues are expected to extend ICMPG’s impact:

• Integration of richer world models for more complex physical interaction and scene-aware reasoning.
• Improved real-time performance through efficient simulator surrogates or hardware acceleration.
• Enhanced multimodal controllability, incorporating vision, audio, and explicit goal signals for embodied agents.

Conclusion

ICMPG establishes a robust methodology for bridging open-vocabulary semantics and physical realism in motion synthesis. By recasting motion generation as receding-horizon, reward-driven optimization over language-planned candidates, it eliminates the need for task-specific policy learning and enables state-of-the-art generalization performance in challenging, diverse settings. This framework offers a foundation for future controllable, adaptable, and physically verifiable motion agents conditioned on naturalistic language.