DMLoco: Diffusion & PPO for Quadruped Locomotion

• The paper introduces DMLoco, a two-stage framework integrating diffusion-based multi-task pretraining with PPO reinforcement learning for robust quadruped control.
• It leverages expert gait demonstrations and natural language conditioning to enable smooth transitions and real-time execution under diverse conditions.
• DMLoco achieves high stability and precise tracking on both simulated and physical platforms, outperforming prior imitation and model-free RL approaches.

DMLoco is a two-stage framework that combines diffusion-based multi-task pretraining and online reinforcement learning for quadruped robot control, enabling robust, language-conditioned locomotion with efficient real-time execution. It addresses the challenges encountered by prior imitation learning and model-free reinforcement learning (RL) solutions, particularly compounded errors in generation, poor task transitions, and limited support for natural language instruction during high-frequency control of legged robots. DMLoco's pipeline leverages a diffusion policy model trained on diverse expert gaits and further finetuned with Proximal Policy Optimization (PPO), facilitating strong generalization, interpretability, and deployment on resource-limited hardware while attaining high stability and tracking performance across both simulation and physical platforms (Qin et al., 8 Jul 2025).

1. Architectural Foundations and Motivation

DMLoco is structured as a dual-phase learning pipeline optimized for adaptive locomotion and task switching in quadruped robots. In the first stage, a Denoising Diffusion Probabilistic Model (DDPM) is trained via behavioral cloning on a dataset of expert demonstrations encompassing four fundamental gaits—trotting, bounding, pacing, and pronking. Each trajectory within this dataset is annotated with velocity/gait commands and free-form natural language descriptions, capturing multi-modal context and enabling language-guided action generation.

The second stage transitions to online RL, specifically PPO-based finetuning. This phase operates on the pretrained diffusion model, exposing the policy to simulated interactions that include out-of-distribution states, unobserved gait transitions, and domain-randomized perturbations. The PPO stage does not require elaborate reward shaping; rather, it uses a simple objective combining velocity tracking and a fall penalty, integrating exploration by modulating diffusion noise schedules. This design addresses high-frequency instability and compounding errors observed in previous applications of diffusion models to locomotion, and directly improves robustness in both simulated environments and real robot deployments (Qin et al., 8 Jul 2025).

2. Multi-Task Diffusion Pretraining

The diffusion model in DMLoco is parameterized by a U-Net architecture tailored for action denoising. At each denoising iteration $k$, the model is fed a noisy chunked action sequence $A_k^t$ (covering a horizon of $h_a$ steps), a recent state history $S_t = s_{t-h_s:t}$, a task embedding $G_t$, and the current step index. The task embedding supports either structured command vectors $[v_x, v_y, \omega, \text{gait one-hot}]$ or the output of a language encoder mapping free-form instructions to gait-aligned representations.

The diffusion objective is formalized as

$$L_{\mathrm{diff}} = \mathbb{E}_{x_0, \epsilon, k} [ \|\epsilon - \epsilon_\theta(x_k, k)\|^2 ],$$

where $x_k$ denotes the noisy action trajectory and $\epsilon_\theta$ the U-Net’s denoising prediction. For language conditioning, a pretrained All-MiniLM-L6-v2 encoder with a 2-layer MLP is used, bringing language vector space into correspondence with gait representations via

$$L_{\mathrm{lang}} = \mathbb{E}_{(l, g)} \left [ \| E_l(l) - g_{\text{onehot}} \|^2 \right ].$$

Joint optimization across both objectives ensures that the model supports both structured and natural language commands without significant degradation in performance (96–97% success with pretrained encoders, versus 76% from-scratch baselines). The pretraining dataset consists of 300 expert rollouts (500 steps each), fully randomized in control gains and environment parameters to promote generalization across gaits and instruction modalities (Qin et al., 8 Jul 2025).

3. PPO Finetuning and Task Transition Robustness

Following diffusion pretraining, the policy undergoes online finetuning with PPO using the simulator (Isaac Gym) as the environment. The diffusion process acts as an inner Markov Decision Process ("denoising MDP") nested within the environment's stepwise dynamics:

• At each decision epoch, the policy performs $A_k^t$0 denoising steps to generate an action $A_k^t$1 conditioned on $A_k^t$2 and $A_k^t$3.
• This action is executed, and the tuple $A_k^t$4 is stored for PPO updates.

PPO employs a clipped surrogate loss with generalized advantage estimation (GAE), value function regularization, and entropy bonus. During finetuning, the number of diffusion steps is reduced (default $A_k^t$5), and the sampling noise schedule is increased to promote exploration and robustness. The policy is specifically trained to navigate both intra-gait and inter-gait (e.g., trotting to pronking) transitions under dynamically shifting environments, significantly advancing transition success rates after fine-tuning (simulation: 91%, real: 75%) compared to initial pretrained models (simulation: 63%, real: 10%) (Qin et al., 8 Jul 2025).

4. Language Conditioning and Model Input Processing

DMLoco natively supports language-conditioned locomotion through its encoder pipeline. Task descriptors supplied in free-form English are embedded using the MiniLM-based encoder, which is co-trained to align its outputs to the one-hot gait command vectors. This mechanism yields language-guided behaviors with near-parity to structured input performance, offering an intuitive and flexible interface for human-robot interaction.

Internally, the action and state representations are chunked with optimal horizons identified via ablation (state horizon $A_k^t$6 steps, action horizon $A_k^t$7 steps). Shorter horizons yield insufficient context; excessively long horizons result in performance decay due to stale data and increased computational overhead.

5. Efficient Inference: DDIM and Deployment Optimization

To meet the real-time and embedded constraints of quadruped robotics, DMLoco deploys Denoising Diffusion Implicit Models (DDIM) for deterministic and highly efficient action sampling. The DDIM update takes the form

$A_k^t$8

enabling high-stability action generation in as few as 5 denoising steps. This sharply contrasts standard DDPM inference, which collapses to instability under aggressive step reduction.

Software and hardware optimizations are achieved by converting the PyTorch model to ONNX and then TensorRT: on Jetson Orin NX, this provides 53 Hz control loop capability, versus $A_k^t$97 Hz in raw PyTorch. This enables fully onboard, high-frequency policy execution required for physical quadruped deployment (Qin et al., 8 Jul 2025).

6. Experimental Results and Ablation Analysis

Empirical evaluations encompass both simulation and real-world tests, using four gaits at randomized velocities and task transitions. Key metrics are stability (percentage of 10 s trials without fall) and mean squared tracking error:

• Pretrained and finetuned DMLoco achieve 98–100% stability with tracking error as low as 0.10–0.31 $h_a$0 depending on gait and phase.
• Gait transition success rate improves from 63% (simulation, pretrained) to 91% (simulation, finetuned); on hardware from 10% to 75%.
• Ablation reveals language encoder selection minimally degrades performance (All-MiniLM-L6-v2 versus BERT: 96% vs 97%), and validates the superiority of DDIM-5-step inference (99% stability at 0.028 s/sample) compared to DDPM-5-step (0%).

The combination of multi-task diffusion pretraining and PPO finetuning outperforms prior methods by capturing the multi-modal nature of gaits, providing robust handling of abrupt transitions, and achieving scalability to hardware-constrained platforms without complex reward tuning (Qin et al., 8 Jul 2025).

7. Comparative Summary and Significance

DMLoco constitutes the first framework to demonstrate that diffusion-based multi-task pretraining, synergistically combined with online PPO finetuning, can enable robust, stable, and language-commanded quadruped locomotion. The system's generalization capability, support for natural language, real-time inference efficiency, and empirical evidence affirm its significance for research and deployment in adaptive robotics. Notably, DMLoco achieves 100% stability over four gaits, high-speed and reliable transitions, and real-time 50 Hz onboard execution across diverse settings (Qin et al., 8 Jul 2025).