Papers
Topics
Authors
Recent
Search
2000 character limit reached

SCRIPT: Scalable Diffusion Policy with Multi-stage Training for Language-driven Physics-Based Humanoid Control

Published 21 May 2026 in cs.GR, cs.LG, and cs.RO | (2605.22894v1)

Abstract: Controlling physics-based humanoids from natural-language instructions is a critical step toward general-purpose embodied agents. However, existing methods remain constrained by a tension between semantic expressiveness and physical feasibility, often failing to jointly achieve faithful instruction following, high-quality motion, and stable long-horizon control. We propose SCRIPT, a scalable diffusion policy with a multi-stage training framework for language-driven physics-based humanoid control. The core of SCRIPT is a Joint Action-State-Text Diffusion Transformer (JAST-DiT), which represents actions, physical states, and text as dedicated token streams and couples them through joint attention, enabling direct interaction between language semantics and control dynamics. To stabilize autoregressive control, we introduce a nonlinear history conditioning mechanism, which preserves the dense recent context and samples increasingly sparse cues from long-term history. Beyond supervised imitation pre-training, we propose a post-training stage, further improving the performance using Reinforcement Learning with Hybrid Rewards (RLHR). By injecting learnable noise into the flow-sampling process, RLHR effectively improves motion quality and instruction following within closed-loop simulations using hybrid physical feedback and text rewards. Quantitative evaluations demonstrate that SCRIPT outperforms prior state-of-the-art methods, with gains across text alignment, motion quality, and physical realism metrics. Furthermore, scaling studies on the 1200-hour MotionMillion dataset demonstrate consistent performance gains with model scaling, highlighting SCRIPT's robust scalability for large-scale pre-training. Our code will be publicly available for future research.

Summary

  • The paper introduces SCRIPT, a scalable diffusion policy that integrates language, state, and action tokens for seamless physics-based humanoid control.
  • The paper employs a two-stage training approach combining supervised imitation with RL post-training using hybrid rewards for enhanced semantic and physical alignment.
  • The paper demonstrates significant improvements in text-motion alignment, physical stability, and motion diversity over state-of-the-art baselines using the JAST-DiT architecture.

SCRIPT: Scalable Diffusion Policy for Language-Driven Humanoid Control

Motivation and Problem Statement

Language-driven control of physics-based humanoids is a pivotal direction for advancing embodied agents that operate autonomously in complex environments. Existing paradigms have struggled to reconcile semantic expressiveness and physical feasibility, often sacrificing instruction fidelity, motion quality, or robustness in closed-loop dynamics. Approaches that decouple kinematic planning from physics execution incur propagation of infeasible references; unified controllers often collapse diverse motion modalities due to mode averaging. The paper introduces SCRIPT, which addresses these compounding gaps with a scalable, end-to-end diffusion policy that tightly couples language, state, and action modeling.

SCRIPT Framework and JAST-DiT Architecture

SCRIPT leverages the Joint Action-State-Text Diffusion Transformer (JAST-DiT), which uniquely represents actions, physical states, and textual instructions as independent token streams. Cross-modal dependencies are modeled through joint attention, allowing token-level interaction and avoiding the information bottlenecks of sentence-level text conditioning. The architecture incorporates nonlinear long-term history conditioning, maintaining dense sampling of recent state transitions while sparsely sampling distant history, which is empirically shown to stabilize autoregressive closed-loop rollouts.

The framework operates in two stages:

  • Stage I: Supervised imitation pre-training via flow matching regression, using curated, physically executable state-action trajectories from large-scale text-motion datasets (e.g., HumanML3D and MotionMillion).
  • Stage II: RL post-training with Hybrid Rewards (RLHR), fine-tuning the policy with reward-driven optimization under stochastic closed-loop simulation in Isaac Gym. Hybrid rewards combine dense physical regularization (penalizing deviation from reference kinematics) and trajectory-level semantic alignment, evaluated via a frozen text-state contrastive model.

Methodological Advancements

Data Curation: Physically valid motion sequences are ensured by filtering for artifact-free kinematic trajectories and discarding sequences with excessive simulation tracking error or humanoid falls. Perturbed rollouts are used to improve robustness to state deviations, exposing the model to off-distribution states during behavioral cloning.

Flow Matching Diffusion: The conditional policy is trained to predict future state-action trajectory chunks, denoising from Gaussian perturbations in accordance with flow matching dynamics, enabling stable trajectory generation.

Nonlinear History Sampling: Inspired by cognitive memory models, the action/state streams attend hierarchically to sparsely sampled distant frames and densely sampled recent context, balancing short-term dynamics with global coherence.

RLHR Optimization: Stochastic noise injection in the action subspace transforms deterministic flow sampling into a Markov chain, supporting tractable policy-gradient updates. Hybrid rewards encourage simultaneous physical stability and semantic faithfulness, with a behavior-cloning anchor to prevent catastrophic forgetting.

Empirical Results

SCRIPT surpasses state-of-the-art baselines in text-motion alignment (R-precision), motion quality (FID, Diversity), and physical stability (Floating, Jerk, Duration time) across HumanML3D and MotionMillion benchmarks. Compared to hierarchical planner-tracker and expert distillation baselines, SCRIPT achieves higher fidelity in prompt following and physical plausibility. Notably, model scaling experiments from 206M to 1.2B parameters show monotonic improvements in all major metrics, evidencing robust utilization of additional capacity.

Ablation studies dissect the contributions of each modality stream, history conditioning, and reward design:

  • Removing text tokens causes semantic collapse; omitting state streams degrades motion stability.
  • Uniform or absent history sampling sharply reduces temporal coherence and semantic matching.
  • Hybrid rewards are essential: Physical-only rewards yield stable but instruction-agnostic motions; text-only rewards preserve alignment but degrade physical realism.

Practical and Theoretical Implications

SCRIPT demonstrates that scalable diffusion policies, tightly coupled with cross-modal joint attention and nonlinear temporal memory, address the fundamental challenge of language-driven closed-loop humanoid control. By directly predicting executable actions in simulation, SCRIPT eliminates reference-tracking mismatch and enables efficient reinforcement-driven adaptation to diverse instructions and environmental feedback.

Practically, SCRIPT is positioned for deployment in virtual avatars, robotics, and simulation environments requiring open-vocabulary motion generation grounded in physical dynamics. The approach promises advances in semantic robustness, generalization, and model scaling, relevant for large-scale embodied AI systems.

Theoretically, the results underscore the necessity of explicit cross-modal token streams, hierarchical memory, and hybrid reward optimization in scaling instruction-conditioned control policies. The positive scaling trends suggest further benefit from increased model capacity and data diversity.

Future Directions

The authors propose extensions toward human-object interaction, multi-agent scenarios, and open-world tasks. Integration with grounding mechanisms, contrastive preference models, and real-world feedback could further enhance generalization and robustness.

Conclusion

SCRIPT introduces a holistic framework for scalable, language-driven, physics-based humanoid control, achieving robust semantic alignment, motion quality, and physical fidelity via the JAST-DiT architecture and RLHR post-training. Empirical results validate the advantages of cross-modal token modeling, scalable architectures, and hybrid optimization, charting a path toward general-purpose embodied agents with compositional instruction-following and physically plausible execution (2605.22894).

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Explain it Like I'm 14

SCRIPT: Teaching a Simulated Humanoid to Follow Language Instructions

What’s this paper about? (Brief overview)

This paper introduces SCRIPT, a new AI system that lets a simulated human body (a “humanoid”) move in a physically realistic way by simply reading natural-language instructions like “walk forward slowly, then turn right.” SCRIPT focuses on making the motions not just match the words, but also obey physics (balance, gravity, friction) and stay stable over long periods.

What questions are the researchers trying to answer?

The team set out to solve four main challenges:

  • Can a single system turn text instructions into body movements that look natural and obey physics?
  • How can the meanings of words connect directly to low-level body control (joint angles, balance, contact with the ground)?
  • How can the system stay stable and consistent for long sequences, not just a few steps?
  • Does making the model bigger and training it on more data keep improving results?

How does SCRIPT work? (Methods explained simply)

The researchers built SCRIPT in three key parts and trained it in two stages.

  1. Large, clean training data
  • They collected lots of human motion clips paired with text descriptions. Many were “kinematic” (just 3D pose data) and not guaranteed to obey physics.
  • To make them usable for a physics-based controller, they filtered out bad clips (like feet going through the floor) and then “tracked” the good ones in a physics simulator to create realistic state-action examples. In the end, they kept about 550,000 physically valid motion sequences, covering around 1,200 hours.
  1. A model that lets text, body state, and actions “talk” to each other
  • SCRIPT uses a transformer model called JAST-DiT (Joint Action-State-Text Diffusion Transformer).
  • Think of three parallel “conversations” running at the same time:
    • Text tokens (words) capture what the instruction says.
    • State tokens describe the body’s current physical condition (joint angles, velocities, etc.).
    • Action tokens represent what the joints should do next.
  • The model uses “joint attention” so these three streams can share information. In simple terms, it lets specific words (like “turn clockwise”) directly influence the exact body parts and movements needed.
  • It also uses a smart memory trick for stability. The model remembers recent steps in detail (because they matter most for balance), and keeps only a few snapshots from farther back. This is like remembering what you just did clearly while keeping a light summary of older context.
  1. Two-stage training that balances imitation and trial-and-error
  • Stage I: Imitation with diffusion/flow matching
    • Diffusion here means learning to turn noisy, messy guesses into clean, realistic motion chunks. “Flow matching” teaches the model the direction to move from noise toward a good trajectory, like learning the wind that pushes a paper airplane toward its destination.
    • The model learns by copying the curated, physically valid examples.
  • Stage II: Reinforcement learning (RL) to polish behavior
    • After imitation, the model practices in the physics simulator, trying actions and getting rewards. It’s guided by a “hybrid” reward:
    • Physics reward: Stay stable, smooth, and realistic.
    • Text reward: Match what the instruction asked for.
    • The model also adds a little controlled randomness to its action generation (like exploring slightly different ways to do the same move), then keeps the ones that score better. This helps it improve without forgetting what it already learned.

Small note on how actions are executed:

  • Actions are target joint angles sent to a PD controller (think of springs and dampers that gently pull joints toward desired positions), which keeps movements smooth and physically believable.

What did they find, and why does it matter?

The researchers tested SCRIPT against strong baselines and found:

  • Better instruction following: Motions matched the text more closely (for example, following direction and style cues like “walk clockwise slowly”).
  • Higher motion quality: Movements looked more natural and less jerky.
  • Stronger physical stability: Fewer glitches like foot sliding or falling; longer stable durations in the simulator.
  • Scales well with size and data: Bigger versions of SCRIPT trained on a huge dataset (MotionMillion) kept improving, handling more diverse actions like walking, sports, and dance while staying physically plausible.

Why this matters:

  • Instead of having a separate “planner” that imagines a move and a “tracker” that tries to copy it (which often leads to mismatches), SCRIPT directly turns text into physically executable actions. This reduces failures and keeps behavior faithful to the instructions.
  • It shows that language, physics, and control can be tightly connected in one model, which is a step toward smarter embodied agents that understand and act in the real world.

What’s the potential impact?

  • Games and animation: Faster creation of lifelike character motions from simple text prompts.
  • Robotics and embodied AI: A path toward robots that can take everyday language and perform tasks safely and smoothly.
  • Research and scaling: Demonstrates that larger models and datasets can steadily improve performance, encouraging future work on more complex scenarios.

The authors suggest next steps like handling objects (human–object interactions), working with multiple agents at once, and operating in richer, open environments.

View Paper Prompt  View All Prompts 

Knowledge Gaps

Below is a consolidated list of concrete knowledge gaps, limitations, and open questions that remain unresolved in the paper. Each item is phrased to enable follow-up investigation by future researchers.

  • Environment and task diversity: The study focuses on single-agent humanoid motion in empty scenes; it does not evaluate uneven terrains, obstacles, variable friction, moving platforms, external perturbations, or partial support (stairs, slopes, stepping stones).
  • Object and tool interaction: No modeling or control of hand/finger actuation or object states; extending to grasping, manipulation, human–object contact modeling, and contact-rich tasks is unaddressed.
  • Multi-agent interactions: Coordination, collision avoidance, and role-dependent behaviors in multi-agent settings are not considered.
  • Real-world transfer: Sim-to-real adaptation, hardware experiments, and robustness to dynamics mismatch (actuator latency, compliance, sensor noise) are not studied.
  • Exteroceptive perception: The policy conditions only on proprioceptive state and text; no visual, depth, or map inputs are used to ground instructions in scene context.
  • Language complexity and compositionality: The model is not evaluated on multi-sentence, temporally compositional, or stepwise instructions (e.g., “walk to the door, then turn left and sit”); online re-prompting mid-episode is unsupported.
  • Cross-lingual generalization: Reliance on CLIP text features leaves multilingual capability and robustness to linguistic variations (negation, quantifiers, spatial deictics) untested.
  • Reference dependence in RL: The physical reward term requires a kinematic reference, limiting applicability to reference-free, goal-only, or open-ended tasks; methods for purely task-based or constraint-based physical rewards remain unexplored.
  • Sparse text reward: The trajectory-level text reward is only given at episode end, creating sparse credit assignment; exploration of dense, step-level semantic rewards or intermediate language-grounded shaping is lacking.
  • Text-reward reliability: The frozen state–text contrastive model’s training data, calibration, and bias are unspecified; risks of reward hacking and misalignment are not analyzed.
  • Human evaluation: No human preference or user study validates whether automatic metrics align with perceived instruction fidelity, style, or appeal.
  • Physics metrics coverage: Physical evaluation uses Floating, Jerk, and Duration only; key measures such as foot slip, ground reaction forces, energy/torque usage, CoM/ZMP stability margins, foot clearance, and contact impulse realism are absent.
  • Robustness stress tests: Generalization to out-of-distribution prompts, morphology changes (link lengths, masses), sensor/actuation noise, and external pushes is not reported.
  • Safety and constraints: No explicit safety constraints (e.g., joint torque/velocity limits in the objective), constraint satisfaction mechanisms, or safe exploration strategies are incorporated or evaluated.
  • Action space limitations: Actions are PD angle targets with fixed gains; alternative actuation (torque/muscle models), adaptive gains, or force/impedance objectives are not explored.
  • History conditioning limits: The nonlinear history sampler improves stability, but sensitivity to Ns, Nl, α, and Lmax is not systematically studied; very long-horizon memory requirements and memory/computation trade-offs remain open.
  • Prediction horizon sensitivity: The impact of the receding-horizon length H on stability, responsiveness, and text adherence is not analyzed.
  • Architectural alternatives: No comparison to causal masking, recurrent memory, KV caching, or hybrid recurrent–transformer designs; joint-attention vs pure cross-attention trade-offs are not examined.
  • Stochastic flow sampling design: The learnable noise schedule σϕ is constrained to a narrow range and conditioned on τ and history, but its stability, exploration–exploitation trade-offs, and ablations (range, schedule, masking variants) are not provided.
  • Scaling analysis depth: Scaling is demonstrated up to ~1.2B parameters, but compute/data efficiency, diminishing returns, scaling laws (error vs model/data/compute), and sample complexity for RL are not quantified.
  • Dataset curation bias: Aggressive filtering and tracking-based selection may bias away from hard or unusual motions; coverage across activities, styles, speeds, and motion complexities is not characterized.
  • Failure mode characterization: The paper lacks a systematic analysis of where SCRIPT fails (e.g., acrobatics, extreme agility, rapid direction changes, unusual stylistic motions) and why.
  • Planner–tracker baselines with feasible plans: Comparisons do not include trackers fed by dynamically feasible, contact-aware planners; the margin vs best-in-class feasible-planner pipelines is unknown.
  • Metric–behavior link: It is unclear how improvements in R-Precision/FID translate to closed-loop behavioral fidelity and user-perceived alignment in physics; establishing causal links between metrics and embodied performance remains open.
  • Runtime and deployment: Inference latency, throughput, and memory footprint for the 0.2B–1.2B models in a real-time control loop are not reported; feasibility for on-robot deployment is uncertain.
  • Continual/online learning: The method does not address continual adaptation, catastrophic forgetting post-RL, or safe policy updates in nonstationary environments.
  • Reward/anchor weighting: The PPO objective relies on hand-tuned weights (BC anchor, entropy, value losses) without principled selection; automated tuning or constrained policy optimization is not investigated.
  • Guidance and conditioning knobs: Effects of classifier-free guidance dropout and guidance scales on instruction fidelity vs diversity are not ablated.
  • Physics engine generality: Portability to different simulators/engines (e.g., MuJoCo, Bullet) and sensitivity to engine-specific contact/solver settings are untested.
  • Body-part grounding: While token-level text features are used, explicit evaluation of grounding to referenced body parts (“raise left knee,” “turn head right”) is missing; fine-grained grounding benchmarks could expose limits.
View Paper Prompt  View All Prompts 

Practical Applications

Immediate Applications

Below are concrete applications that can be deployed now or with minimal engineering, derived from SCRIPT’s findings (JAST-DiT architecture, nonlinear history conditioning, RL post-training with hybrid rewards, scalable data curation). Each entry highlights the sector, potential products/tools/workflows, and feasibility assumptions.

  • Sector: Software/Content Creation (Games, Film, XR)
    • Application: Text-to-physics-consistent character animation for previsualization and in-engine prototyping (e.g., “do a spinning dance,” “crouch then sprint”).
    • Tools/Products/Workflows: Unity/Unreal plugin or SDK that wraps SCRIPT as a local/cloud service; timeline tool to sequence natural-language prompts; batch generation pipeline for designers.
    • Assumptions/Dependencies: Runs in simulation; requires GPU for inference (esp. Large/Huge models); CLIP-based text encoder; no guaranteed realtime on consumer hardware without model distillation/quantization.
  • Sector: Games/Interactive Media
    • Application: Rapid authoring of diverse, physically plausible NPC behaviors from text prompts; augment behavior trees with language-driven motion clips that respect physics (reduced foot-skate, better contacts).
    • Tools/Products/Workflows: Asset generation tool integrated into build systems; prompt libraries for locomotion/sport/dance; QA pipelines with physics metrics (Floating/Jerk).
    • Assumptions/Dependencies: Offline/precompute preferred for performance; designer-in-the-loop review; licensing for MotionMillion/HumanML3D-derived assets.
  • Sector: Robotics R&D (Simulation)
    • Application: Benchmarking and training language-conditioned humanoid policies in Isaac Gym or similar simulators using the multi-stage training recipe (flow-matching pretrain + PPO hybrid rewards).
    • Tools/Products/Workflows: Reproducible training scripts; RL fine-tuning with state-text rewards; rollout filtering/curriculum; experiment dashboards for R-Precision/FID/physics metrics.
    • Assumptions/Dependencies: High-throughput simulation; parallel environments (≥128); curated, physics-executable demonstrations from text-motion datasets.
  • Sector: Robotics Software (ROS/ROS2, Digital Twins)
    • Application: Language-to-action control of digital twins for task rehearsal and operator training (e.g., “walk to the right, then sit and stand”).
    • Tools/Products/Workflows: ROS package that exposes a prompt->action API; teleoperation assist that converts high-level voice/text instructions to low-level PD targets; logging for safety replay.
    • Assumptions/Dependencies: Remains in digital twin/sim; requires calibrated robot model and PD gains; no direct sim-to-real guarantees.
  • Sector: HCI/Virtual Avatars (Social VR, Live Streaming)
    • Application: Prompt-driven avatar motion with improved physical plausibility (dance, gestures, athletic actions) to enhance presence and reduce uncanny artifacts.
    • Tools/Products/Workflows: Streaming overlay that translates chat prompts to motion; avatar middleware for VRChat/VTubing platforms; safety filter for disallowed prompts.
    • Assumptions/Dependencies: Latency budget managed via receding-horizon execution; moderation on language inputs; content policy compliance.
  • Sector: Education/Training
    • Application: Interactive demonstration of physical skills from text (e.g., PE, dance basics) in physics simulation for pedagogy and coaching apps.
    • Tools/Products/Workflows: Prompt playlists; side-by-side motion comparisons; automatic feedback using physics metrics and text alignment scores.
    • Assumptions/Dependencies: Didactic content requires human curation; not a substitute for expert instruction; careful phrasing to avoid unsafe motions.
  • Sector: Data Engineering for Motion/Control
    • Application: Deployable pipeline to transform large kinematic text-motion datasets into physics-executable demonstrations via pre-/post-tracking filtering (artifact rejection, PHC rollout filtering).
    • Tools/Products/Workflows: Data curation toolkit with artifact detectors; expert tracking policy farm; quality gates using physical metrics and failure heuristics.
    • Assumptions/Dependencies: Access to original datasets; compute for large-scale tracking; license compliance and privacy considerations.
  • Sector: ML Infrastructure
    • Application: Adoption of nonlinear history conditioning and joint action-state-text modeling for other long-horizon control tasks (e.g., legged locomotion, manipulation in sim).
    • Tools/Products/Workflows: Reference implementation of JAST-DiT blocks; ablation-backed default history sampler; eval harness for stability and instruction fidelity.
    • Assumptions/Dependencies: Task-specific state/action spaces; re-tuning decay parameters; domain reward shaping.
  • Sector: A/B Testing and Model Evaluation
    • Application: Objective QA of motion generators using SCRIPT’s metrics bundle (R-Precision, FID, MM-Dist + Floating/Jerk/Duration) as acceptance criteria.
    • Tools/Products/Workflows: Continuous integration step that rejects prompts with poor alignment or excessive artifacts; regression dashboards.
    • Assumptions/Dependencies: Access to text-motion evaluator; alignment with studio quality bars.
  • Sector: Safety Research (Pre-Deployment)
    • Application: Closed-loop safety probes for language-conditioned motion in simulation (stress prompts, abrupt changes) before real robot trials.
    • Tools/Products/Workflows: Adversarial prompt suites; auto-detection of instability/falls; policy gating by physical metrics thresholds.
    • Assumptions/Dependencies: Simulation fidelity adequate for risk triage; interpretability tooling for failure analysis.
  • Sector: Tooling for RL with Diffusion Policies
    • Application: Use SCRIPT’s stochastic flow sampling with learnable noise head to improve exploration in continuous control tasks.
    • Tools/Products/Workflows: Plug-in module for PPO/SAC training loops; masks for action-only noise; BC anchor integration to prevent forgetting.
    • Assumptions/Dependencies: Task rewards defined; hyperparameter tuning for exploration scale bounds.
  • Sector: Academic Research
    • Application: Reproducible baseline for studying scaling laws in language-conditioned control (0.2B–1.2B variants) and text-token granularity effects.
    • Tools/Products/Workflows: Model zoos; prompt suites; standard training curricula; public project codebase.
    • Assumptions/Dependencies: GPU cluster access; dataset availability; consistent evaluation seeds.

Long-Term Applications

These opportunities require further research, sim-to-real transfer, hardware integration, and/or large-scale validation before deployment.

  • Sector: Service Robotics (Home, Hospitality, Retail)
    • Application: Voice/text-instructable humanoids performing daily-living tasks (e.g., “stand up, walk to the table on the right, pick up cup”).
    • Tools/Products/Workflows: Onboard language interface; JAST-DiT control stack with perception; task planners layered over low-level controller; safety supervisor.
    • Assumptions/Dependencies: Reliable perception and grounding; robust contact and manipulation; sim-to-real transfer; certification and safety compliance.
  • Sector: Healthcare/Assistive Tech
    • Application: Natural-language-controlled rehabilitation/assistive robots and exoskeletons that execute physically safe, smooth motions aligned with therapist instructions.
    • Tools/Products/Workflows: Clinician-in-the-loop programming via prompts; personalized controllers; fall/instability detection; audit logs.
    • Assumptions/Dependencies: Medical-grade safety, biomechanical constraints, personalization data; strict regulatory approval; human-oversight requirements.
  • Sector: Industrial/Logistics
    • Application: Language-guided humanoids for structured tasks (inspection, simple pick-place with locomotion), coordinating with human workers by verbal commands.
    • Tools/Products/Workflows: Integration with MES/WMS; semantic wayfinding; hybrid rewards extended to task KPIs; multi-policy fallback strategies.
    • Assumptions/Dependencies: Robustness to clutter and domain shifts; dexterous manipulation; reliability/MTBF targets; union/safety policy alignment.
  • Sector: Entertainment/Live Events
    • Application: On-stage animatronics/robots that respond to real-time prompts with physically believable actions and choreographies.
    • Tools/Products/Workflows: Low-latency deployment with distilled models; choreographer control panels; motion safety envelopes.
    • Assumptions/Dependencies: Realtime constraints on embedded hardware; redundancy and fault tolerance.
  • Sector: Education/Coaching at Scale
    • Application: Personalized motion coaching (dance, fitness, rehab homework) with natural-language prompts and physically grounded demonstrations.
    • Tools/Products/Workflows: Curriculum generation; progress tracking via physics/stability metrics; multimodal feedback (text + motion).
    • Assumptions/Dependencies: Accurate user pose capture; tailoring for age/ability; liability and content moderation.
  • Sector: Multi-Agent Collaboration
    • Application: Language-directed coordination among multiple humanoids for team tasks (e.g., “two agents carry the box, third opens the door”).
    • Tools/Products/Workflows: Extended JAST-DiT with agent tokens; inter-agent communication; group-level hybrid rewards.
    • Assumptions/Dependencies: Scalable training data for multi-agent dynamics; collision/contact handling; latency constraints.
  • Sector: Human-Object Interaction and Manipulation
    • Application: Rich object-centric tasks (tool use, sports skills) driven by open-vocabulary instructions.
    • Tools/Products/Workflows: Object-aware state tokens; object-text alignment; contact-rich reward shaping; sim assets for diverse objects.
    • Assumptions/Dependencies: High-fidelity contact and friction modeling; grasp planning; perception for 6D object state.
  • Sector: Cross-Embodiment and Hardware Portability
    • Application: Transfer of language-driven control to different humanoid morphologies via residual adapters or morphological tokens.
    • Tools/Products/Workflows: Retargeting pipelines; embodiment-conditioned JAST-DiT; auto-tuning PD gains; domain randomization.
    • Assumptions/Dependencies: Robust sim-to-real bridges; extensive calibration; safety-testing per embodiment.
  • Sector: Standardization and Policy
    • Application: Industry standards for language-to-action safety, alignment testing, and dataset curation for embodied agents.
    • Tools/Products/Workflows: Certification suites using SCRIPT-style metrics; prompt-risk taxonomies; dataset provenance and artifact filters.
    • Assumptions/Dependencies: Multi-stakeholder consensus; regulatory frameworks; third-party test labs.
  • Sector: Cloud Robotics/Control-as-a-Service
    • Application: Prompt-to-motion cloud APIs powering fleets of robots/avatars with centralized updates and monitoring.
    • Tools/Products/Workflows: Managed inference endpoints; usage-based billing; safety and prompt moderation layer; telemetry dashboards.
    • Assumptions/Dependencies: Network latency budgets; privacy/security controls; edge fallback.
  • Sector: Foundation Models for Embodied AI
    • Application: Large-scale pretraining of generalized, language-driven control policies spanning locomotion, manipulation, and social gestures, leveraging SCRIPT’s scaling behavior.
    • Tools/Products/Workflows: Multi-domain datasets; mixture-of-experts JAST-DiT variants; preference/RLHF for physical alignment.
    • Assumptions/Dependencies: Very large compute; comprehensive datasets; robust safety alignment across tasks.
  • Sector: Voice-Controlled Consumer Robotics
    • Application: Household robots responding to conversational commands for mobility tasks (fetch, follow, gesture).
    • Tools/Products/Workflows: On-device distilled controller; edge ASR + NLU; safety geofencing; parent/elder modes.
    • Assumptions/Dependencies: Cost-effective hardware; long-term reliability; clear UX for ambiguity resolution.

Cross-cutting Assumptions and Dependencies

  • Data: Access to large, legally usable, and curated physics-executable text-motion datasets; continued improvement in artifact filtering and tracking.
  • Compute: Training and inference resources scale with model size (0.2B–1.2B+ parameters); potential need for distillation/quantization for edge deployment.
  • Perception and Grounding: Real-world deployments require robust perception (vision, IMU, tactile) and scene grounding, not covered in the paper.
  • Safety and Alignment: Hybrid rewards and state-text alignment help in sim, but real-world safety requires additional constraints, supervision, and certification.
  • Sim-to-Real Gap: Policies trained in Isaac Gym require domain randomization, system ID, and hardware-aware control for transfer.
  • Latency: Receding-horizon control may need optimization for strict realtime constraints; history conditioning must be efficient.
  • IP/Policy: Content moderation, prompt filtering, and ethical use policies for language-driven motion are required for products.
View Paper Prompt  View All Prompts 

Glossary

  • 6D rotation: A continuous rotation representation using two 3D orthogonal vectors, avoiding singularities of Euler angles; commonly used for joint orientations. "6D rotation"
  • actuation limits: Physical bounds on the torques/angles a controller can command, imposed by hardware or simulator constraints. "actuation limits"
  • AdaLN-Zero modulation: An adaptive LayerNorm variant with zero-initialized scaling that conditions features on auxiliary inputs for stable conditioning. "via AdaLN-Zero modulation"
  • autoregressive control: A control strategy that generates each action conditioned on previous states/actions in sequence. "To stabilize autoregressive control, we introduce a nonlinear history conditioning mechanism"
  • behavior cloning: Supervised learning that imitates expert demonstrations by regressing actions from states. "pre-trains a flow matching diffusion policy via behavior cloning"
  • behavior cloning anchor loss: A regularization term anchoring a policy during RL to its pre-trained behavior-cloned outputs to prevent drift. "we augment the standard RL objective with a behavior cloning anchor loss"
  • classifier-free guidance: A technique that mixes conditional and unconditional model predictions (often via text-dropout) to control conditioning strength. "10% text-condition dropout for classifier-free guidance"
  • CLIP pooled embedding: A single vector summary from CLIP’s text encoder representing an entire prompt, used as a global condition. "a single CLIP pooled embedding"
  • closed-loop execution: Control where actions are continually updated based on observed states, forming a feedback loop. "physics-based humanoid control requires closed-loop execution"
  • contact-rich simulations: Physics simulations with frequent contacts (e.g., feet-ground), requiring robust handling of impacts and friction. "contact-rich simulations"
  • cosine similarity: A similarity measure between vectors based on the cosine of the angle between them, used here for text–state alignment. "compute the cosine similarity"
  • cross-attention: An attention mechanism where one sequence (e.g., actions) attends to another (e.g., history) to fuse information. "through cross-attention"
  • Diffusion Transformer (DiT): A Transformer architecture tailored for diffusion/flow models, often using conditioning layers for generative tasks. "the scaling principles of Diffusion Transformers (DiT)"
  • Euler integration step: A first-order numerical integration step used to discretize continuous-time dynamics or flows. "at each Euler integration step"
  • FID (Fréchet Inception Distance): A distributional distance metric assessing generative quality by comparing feature statistics of generated vs. real data. "R-Precision (Top-1/2/3), FID, and MM-Dist"
  • flow-sampling process: The numerical procedure that integrates the learned flow/velocity field to transform noise into samples. "By injecting learnable noise into the flow-sampling process"
  • Flow Matching (FM): A generative modeling framework that learns a velocity field to transport noise to data along a probability path. "based on Flow Matching (FM)"
  • flow time: The continuous scalar parameter τ indicating progression along the probability path in a flow-based model. "where τ is the flow time"
  • Isaac Gym: A GPU-accelerated physics simulation platform for large-scale reinforcement learning and control. "in Isaac Gym simulator"
  • JAST-DiT (Joint Action-State-Text Diffusion Transformer): An architecture with separate token streams for actions, states, and text that interact through attention. "Joint Action-State-Text Diffusion Transformer (JAST-DiT)"
  • jerk: The time derivative of acceleration; used as a smoothness metric for motion. "Floating and Jerk are computed from global joint positions."
  • joint attention: Attention computed over concatenated multi-modal tokens (actions, states, text) to enable dense cross-modal interactions. "couples them through joint attention"
  • kinematic references: Precomputed motion trajectories without enforcing physical dynamics, often used as targets for tracking. "kinematic references"
  • latent skill spaces: Low-dimensional embeddings capturing reusable motor behaviors for compositional control. "latent skill spaces"
  • Markov chain: A stochastic process with memoryless transitions; here, the discretized flow sampler becomes a Markov chain over steps. "turns the deterministic flow sampler into a Markov chain"
  • MM-Dist: A motion–motion distance metric used to assess generated motion quality relative to references. "R-Precision (Top-1/2/3), FID, and MM-Dist"
  • motion priors: Learned models capturing typical human motion patterns used to regularize or guide generation/control. "motion priors"
  • nonlinear long-term history conditioning: A memory mechanism that samples recent states densely and distant states sparsely to stabilize long-horizon rollouts. "a nonlinear long-term history conditioning mechanism"
  • penultimate-layer token features: The sequence of token embeddings from the second-to-last layer of a text encoder, retaining fine-grained semantics. "penultimate-layer token features"
  • planner-tracker design: A two-stage pipeline where a kinematic planner proposes motion and a controller tracks it under physics. "its planner-tracker design remains vulnerable to tracking mismatch"
  • Proportional-derivative (PD) controller: A joint-level controller that computes torques/targets from proportional and derivative terms relative to desired angles/velocities. "a proportional-derivative (PD) controller"
  • PPO (Proximal Policy Optimization): A policy-gradient RL algorithm that stabilizes updates via clipped objectives and advantage estimation. "via PPO"
  • proprioceptive state: Internal body state measurements (e.g., joint angles, velocities) used by a controller. "per-frame proprioceptive state"
  • QK-Norm: Normalization applied to attention queries and keys to improve numerical stability and scaling in Transformer attention. "we apply QK-Norm"
  • receding-horizon manner: An execution scheme predicting a multi-step plan but applying only the first action before replanning. "operates in a receding-horizon manner"
  • Reinforcement Learning with Hybrid Rewards (RLHR): An RL fine-tuning regime combining physics-based and text-alignment rewards. "Reinforcement Learning with Hybrid Rewards (RLHR)"
  • rigid-body dynamics: Physics modeling that treats bodies as rigid and simulates their motion and contacts. "under rigid-body dynamics"
  • R-Precision: A retrieval-based metric evaluating how well generated motions match their conditioning texts among distractors. "R-Precision (Top-1/2/3)"
  • SMPL: A parametric human body model widely used for representing human shape and pose. "SMPL-like"
  • state-text contrastive reward: A reward computed by contrasting encoded trajectories against text prompts to encourage semantic alignment. "a sparse trajectory-level state-text contrastive reward"
  • stochastic flow-sampling trajectories: Rollouts generated by adding noise during flow sampling to enable exploration for RL. "on stochastic flow-sampling trajectories"
  • token streams: Separate sequences of tokens per modality (e.g., action, state, text) processed in parallel within a model. "dedicated token streams"
  • vector field: The learned function that maps each intermediate point and time to a velocity directing the flow toward data. "FM trains a vector field"
View Paper Prompt  View All Prompts 

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up

Tweets

Sign up for free to view the 4 tweets with 87 likes about this paper.

Sign Up for Free