In-Context Model Predictive Generation

• In-Context Model Predictive Generation (ICMPG) is a framework for text-driven human motion synthesis that reformulates motion generation as a receding-horizon control problem using discrete motion tokens.
• It employs a two-module system where the Context-Aware Motion Generation (CAMG) module decomposes instructions and the Model Predictive Generation (MPG) module evaluates candidates via physical simulation and semantic scoring.
• The framework achieves improved physical plausibility and semantic fidelity through a closed-loop, MPC-like iterative refinement process validated by extensive quantitative benchmarks.

Searching arXiv for the cited papers to ground the article. I’ll look up the arXiv entries for the main and related papers mentioned. In-Context Model Predictive Generation (ICMPG) is a framework for text-driven human motion synthesis that integrates language-model planning with inference-time physical feedback, introduced in 2026 as a method for addressing the persistent trade-off between semantic fidelity and physical realism in motion generation (Fu et al., 25 Jun 2026). It reformulates motion synthesis as a Model Predictive Control (MPC)-like process with two modules: Context-Aware Motion Generation (CAMG), which uses an LLM to decompose instructions and propose motion-token candidates, and Model Predictive Generation (MPG), which evaluates those candidates through physical simulation and semantic alignment before selecting the sequence that guides subsequent generation. In the formulation of the paper, this closed-loop refinement allows adaptation to both input semantics and the simulated physical environment without task-specific policy retraining (Fu et al., 25 Jun 2026).

1. Problem setting and conceptual motivation

ICMPG is motivated by a specific failure mode in text-to-motion generation: methods centered on LLMs can interpret diverse open-vocabulary instructions and compose high-level action plans, but they often generate motions that violate physical constraints, whereas physics-aware models improve realism through simulation or control but struggle with semantic complexity, fine-grained instructions, and novel concepts (Fu et al., 25 Jun 2026). The framework is presented as a direct response to that gap.

The central conceptual move is to treat motion generation not as a one-shot decoding problem but as a receding-horizon decision process over discrete motion tokens. CAMG proposes candidate continuations, and MPG evaluates them against two criteria that the paper treats jointly: physical plausibility and semantic faithfulness. This differs explicitly from open-loop generation. The paper states that unlike open-loop generation, the closed-loop refinement enables ICMPG to adapt motions to both the input semantics and the simulated physical environment, while remaining flexible enough to incorporate different LLM backbones (Fu et al., 25 Jun 2026).

A common misunderstanding is to regard ICMPG as a prompt-engineering variant of text-to-motion. In the formulation of the paper, that is too narrow. The LLM is frozen and used as a planner/generator, but the decisive selection step occurs through simulation-time feedback. The resulting system is therefore neither a purely linguistic planner nor a purely physics-based controller; it is an MPC-like synthesis loop over tokenized motion sequences.

2. System architecture: CAMG and MPG

The pipeline consists of two major modules, preceded by one-time offline preparation (Fu et al., 25 Jun 2026). The offline stage includes a VQ-VAE trained to encode 3D human motion clips into discrete motion tokens, a lightweight Transformer called SemAligner trained by contrastive loss to retrieve the top–K text–motion demonstration pairs, and a pre-trained physics policy $\pi$ such as PHC for later simulation.

Within CAMG, the online process begins with an LLM Planner. It is prompted with the raw text $t$ and a small set of atomic “CMU” motion examples $A_{dem}$, and outputs a decomposition $t_p$ into a sequence of simple motion-focused instructions of the form “he does X, he does Y …” (Fu et al., 25 Jun 2026). SemAligner then encodes $t_p$ and retrieves $S_{dem}$, the top–K semantically closest full motion–text pairs from the training set. Finally, the LLM Generator receives $t_p$, $S_{dem}$, and any previously selected motion prefix $o$, and samples $N$ candidate motion-token sequences, each of length $t$0.

The notation given for candidate generation is

$t$1

where each sampled sequence is a token block proposed at iteration $t$2 (Fu et al., 25 Jun 2026). The architecture is therefore compositional in two senses: the text is decomposed into simpler action units, and the motion is built incrementally as a sequence of token blocks chosen under feedback.

The second module, MPG, performs the control-like evaluation. At each iteration it simulates each candidate under policy $t$3 in Isaac Gym, computes a discounted physical return, renders key-frames as SMPL meshes for semantic scoring with ActionCLIP or CLIP, forms a composite reward, and selects the highest-scoring candidate to append to the current history (Fu et al., 25 Jun 2026). In the paper’s summary phrase, CAMG proposes, MPG evaluates and controls, in a receding-horizon MPC-like loop.

3. MPC-like formulation over motion tokens

The paper explicitly casts motion synthesis as a discrete-time control problem over motion tokens (Fu et al., 25 Jun 2026). Let $t$4 denote the current motion-token prefix and let $t$5 denote the next block of $t$6 tokens. The state transition is defined by concatenation: $t$7

This formulation treats each candidate token block as a control action. The short-horizon optimization then amounts to selecting the block that maximizes a composite reward conditioned on the current prefix. The control law is written as

$t$8

followed by the update $t$9 (Fu et al., 25 Jun 2026).

The composite reward is presented in two closely related forms in the paper. In the high-level module description, MPG computes

$A_{dem}$0

In the explicit MPC-like optimization section, the combined score is written as

$A_{dem}$1

with typical choice $A_{dem}$2 (Fu et al., 25 Jun 2026). The paper’s ablation statement that reward weight $A_{dem}$3 trades off R-Precision versus Phys-Err, with $A_{dem}$4 best overall, reinforces the interpretation that ICMPG is tuned as a hybrid semantic-physical objective rather than as a pure feasibility filter.

This MPC analogy should not be overstated. The “state” is a token prefix rather than a continuous physical state, and the “control” is a candidate token block. Nevertheless, the receding-horizon structure, iterative replanning, and reward-based candidate selection are all explicit parts of the formulation (Fu et al., 25 Jun 2026).

4. Prompting, retrieval, and inference-time feedback

ICMPG uses two distinct prompt templates (Fu et al., 25 Jun 2026). The planner prompt asks the model to re-phrase the raw user input into a JSON object with keys "CoT" and "Motions", using only verbs from the atomic actions set and illustrated with 3–5 short atomic examples $A_{dem}$5. The generator prompt asks the model to map the decomposed prompt $A_{dem}$6 plus the prefix tokens $A_{dem}$7 into a list of candidate motion-token sequences, again in JSON format with "CoT" and "Candidate motion tokens" fields. Sampling is performed with temperature $A_{dem}$8 and top-p $A_{dem}$9 (Fu et al., 25 Jun 2026).

The retrieval component is semantically targeted. The few-shot context contains both atomic-action examples and retrieved semantic demonstrations. Specifically, the prompts interleave $t_p$0, the retrieved text–motion pairs $t_p$1, the current decomposition $t_p$2, and the already selected motion prefix $t_p$3 (Fu et al., 25 Jun 2026). This makes the in-context signal heterogeneous: part of it is symbolic decomposition guidance, and part of it is demonstration-based retrieval.

The physical evaluation loop is equally specific. The simulator is NVIDIA Isaac Gym at $t_p$4 Hz, and the policy $t_p$5 is a pre-trained PHC policy that can track any realistic human motion clip (Fu et al., 25 Jun 2026). The physical reward over horizon $t_p$6 is defined as

$t_p$7

where $t_p$8 is the normalized RL reward (Fu et al., 25 Jun 2026). The semantic reward is formed by rendering a subset of key-frames $t_p$9 as SMPL meshes, encoding the text with $t_p$0 and the images with $t_p$1, and computing a cosine-style alignment score: $t_p$2

The closed-loop refinement algorithm then proceeds by repeatedly generating candidates, simulating each one, computing $t_p$3, $t_p$4, and $t_p$5, selecting the best candidate $t_p$6, appending it to the prefix, and decoding the final token sequence through the VQ-VAE decoder into 3D motion (Fu et al., 25 Jun 2026). This is the operational core of ICMPG: inference is itself the control loop.

5. Experimental results and ablations

The paper reports extensive experiments across standard and zero-shot open-vocabulary settings, and states that ICMPG generalizes robustly to diverse commands while producing motions that are more physically plausible and semantically faithful than representative baselines on the evaluated benchmarks (Fu et al., 25 Jun 2026). The quantitative profile is divided across BABEL, HumanML3D, KIT-ML, and zero-shot generalization benchmarks.

On the zero-shot BABEL evaluation, with no BABEL training unless otherwise noted, the reported FID values are: AvatarCLIP $t_p$7, MotionCLIP $t_p$8, AnySkill $t_p$9, ICMPG (no LoRA) $S_{dem}$0, and ICMPG (LoRA) $S_{dem}$1, with the LoRA variant marked as best (Fu et al., 25 Jun 2026). For CLIP_Score, the reported values are AvatarCLIP $S_{dem}$2, MotionCLIP $S_{dem}$3, AnySkill $S_{dem}$4, ICMPG $S_{dem}$5, and ICMPG (LoRA) $S_{dem}$6, again best for the LoRA variant. For R-Precision@1, MotionCLIP obtains $S_{dem}$7, AnySkill $S_{dem}$8, ICMPG $S_{dem}$9, and ICMPG (LoRA) $t_p$0. On simulation success rate, AnySkill reaches $t_p$1, ICMPG $t_p$2, and ICMPG (LoRA) $t_p$3. For Phys-Err, the values are AnySkill $t_p$4, ICMPG $t_p$5, and ICMPG (LoRA) $t_p$6 (Fu et al., 25 Jun 2026).

On standard text-to-motion benchmarks, the paper reports for HumanML3D that R-Precision@1 is $t_p$7 for T2M-GPT, $t_p$8 for MotionDiffuse, $t_p$9 for ICMPG, and $S_{dem}$0 for ICMPG(LoRA), while Phys-Err is $S_{dem}$1 for MotionDiffuse, $S_{dem}$2 for T2M-GPT, $S_{dem}$3 for ICMPG, and $S_{dem}$4 for ICMPG(LoRA) (Fu et al., 25 Jun 2026). For KIT-ML, the listed R-Precision@1 values are MDM $S_{dem}$5, T2M-GPT $S_{dem}$6, ICMPG $S_{dem}$7, and ICMPG(LoRA) $S_{dem}$8, while Phys-Err is MDM $S_{dem}$9, T2M-GPT $o$0, ICMPG $o$1, and ICMPG(LoRA) $o$2 (Fu et al., 25 Jun 2026).

On zero-shot generalization with MotionMillion and PP-Motion, semantic performance measured by MM-Hit@1 is $o$3 for MotionGPT, $o$4 for MotionCraft, and $o$5 for ICMPG, marked as best (Fu et al., 25 Jun 2026). For physical metrics, UniPhys attains Mod-Cost $o$6 and Penetration $o$7, identified as best physics, MotionCraft gives $o$8, and ICMPG gives $o$9, described as a strong hybrid (Fu et al., 25 Jun 2026).

The ablations are structurally important. The paper states that reward weight $N$0 trades off R-Precision and Phys-Err, with $N$1 best overall; larger LLM backbones, specifically LLaMA-3.1 and Phi-4, improve semantic scores; and inclusion of the LLM Planner module drastically improves FID and success on long, complex prompts (Fu et al., 25 Jun 2026). This suggests that the planner is not merely an auxiliary prompt formatter, but a major determinant of long-horizon compositional behavior.

6. Broader usage of the term and relation to adjacent work

Within the motion-synthesis literature, ICMPG refers specifically to the CAMG–MPG framework of (Fu et al., 25 Jun 2026). At the same time, several related papers and expository mappings use “In-Context Model Predictive Generation” as a broader label for systems that combine in-context inference with receding-horizon prediction, feedback, or control. This suggests that the acronym is functioning both as a proper name for a specific framework and as a more general design pattern.

In dynamic manipulation, Van der Merwe and Jha describe an in-context iterative policy-improvement loop that can be rendered in MPC-style ICMPG form: a parametric policy $N$2 is rolled out, a task-specific error $N$3 is computed, the $N$4 nearest examples are retrieved from an offline dataset, an LLM predicts $N$5, and the update $N$6 is applied for $N$7 iterations; the reported retrieval size is $N$8, the model is gpt-4o with temperature $N$9 and top_p $t$00, and the final best-policy cost after 20 iterations reaches $t$01 on slide and $t$02 on roll-gc-real (Merwe et al., 20 Aug 2025). In that usage, ICMPG denotes LLM-mediated receding-horizon policy adjustment rather than token-level motion synthesis.

In power-grid control, InstructMPC presents a closed-loop framework in which an LLM-guided Contextual Disturbances Predictor produces disturbance forecasts for finite-horizon MPC, and only the decoder’s last layer is fine-tuned online; for linear systems optimized via the tailored loss, the paper gives a regret bound of $t$03 (Wu et al., 5 Dec 2025). In prompt optimization, self-instructed derived-prompt generation is explicitly interpreted as a special case of a more general ICMPG paradigm in which a planner proposes a derived prompt, a frozen response model and reward model evaluate it, and the resulting pair $t$04 is re-used as an in-context demonstration (Li et al., 2024). In language-model pretraining, ContextLM is described as realizing ICMPG by predicting next-context embeddings and feeding them back into token decoding; for GPT2-XL the average perplexity is reported as approximately $t$05 (Dai et al., 23 Oct 2025).

These broader usages matter because they clarify what is distinctive about the 2026 motion-synthesis framework and what is more generic. The specific contribution of (Fu et al., 25 Jun 2026) is not simply that it is “closed-loop” or “in-context,” since related work uses similar descriptors in other domains. Its specificity lies in the coupling of open-vocabulary motion planning, discrete motion tokens, simulation-based physical evaluation, semantic scoring from rendered SMPL key-frames, and iterative selection under an MPC-like loop. A plausible implication is that ICMPG names both a concrete architecture for text-driven human motion synthesis and an emerging cross-domain pattern for combining LLM proposals with predictive feedback.