TransCurriculum: Multi-Dimensional Curriculum Learning for Fast & Stable Locomotion

Abstract: High-speed legged locomotion struggles with stability and transfer losses at higher command velocities during deployment. One reason is that most curricula vary difficulty along single axis, for example increase the range of command velocities, terrain difficulty, or domain parameters (e.g. friction or payload mass) using either fixed update rule or instantaneous rewards while ignoring how the history of robot training has evolved. We propose TransCurriculum, a transformer-based multi-dimensional curriculum learning approach for agile quadrupedal locomotion. TransCurriculum adapts to 3 axes, velocity command targets, terrain difficulty, and domain randomization parameters (friction and payload mass). Rather than feeding task reward history directly into the low-level control policy, our formulation exploits it at the curriculum level. A transformer-based teacher retrieves the sequence of rewards and uses it to predict future rewards, success rate, and learning progress to guide expansion of this multidimensional curriculum towards high performing task bins. Finally we validate our approach on the Unitree Go1 robot in simulation (Isaac Gym) and deploy it zero-shot on Go1 hardware. Our TransCurriculum policy achieves a maximum velocity of 6.3 m/s in simulation and outperforms prior curriculum baselines. We tested our TransCurriculum trained policy on terrains (carpets, slopes, tiles, concrete), achieving a forward velocity of 4.1 m/s on carpet surpassing the fastest curriculum methods by 18.8% and achieves maximum zero-shot value among all tested methods. Our multi-dimensional curriculum also reduces the transfer loss to 18% from 27% for command only curriculum, demonstrating the benefits of joint training over velocity, terrain and domain randomization dimension while keeping the task success rate of 80-90% on rigid indoor and outdoor surfaces.

• The paper introduces a transformer-guided curriculum that schedules commands, terrain, and randomized dynamics jointly, using local reward history across 4,000 task bins to expand training efficiently.
• TransCurriculum reaches 6.3 m/s in simulation and 4.1 m/s zero-shot on a Unitree Go1, achieving 90% task success on carpet and outperforming comparable curriculum methods.
• Multidimensional scheduling improves robustness and reduces sim-to-real transfer loss from 27% to 18%, while ablations show history modeling is essential for high-speed learning and stability.

Motivation and problem statement

Reinforcement learning (RL) policies for high-speed quadrupedal locomotion degrade when commanded velocities, terrain properties, and unmodeled dynamics interact at deployment. The authors identify a structural cause in existing curriculum learning approaches: most curricula vary difficulty along a single axis—command velocity, terrain difficulty, or domain randomization (DR) parameters such as friction and payload mass—using fixed or threshold-based update rules. Because these factors interact in the real world, a policy trained with a command-only curriculum can track high velocities on high-friction ground but fails when friction drops. A second limitation is that single-axis schedulers cannot detect temporal trends in policy learning and may repeatedly sample already-mastered task regions.

TransCurriculum addresses both issues by treating curriculum design as a multi-dimensional scheduling problem over the joint task space of commands $c$, domain parameters $d$, and terrain difficulty $t$, with task context $z = [c, d, t]$. Its distinguishing feature is where history is used: rather than feeding reward history into the low-level control policy, the method exploits it exclusively at the curriculum level, so performance gains are attributable to task scheduling rather than additional policy inputs.

Method

The joint task space is discretized into a grid of bin centroids; the experiments use $20 \times 10 \times 20 = 4000$ bins spanning normalized commands, friction $\mu$, payload mass $m$, and terrain difficulty $t \in [0,1]$. Each bin carries a weight $w_i$, and tasks are sampled proportionally to these weights, with specific contexts drawn uniformly within the selected cell.

Three components drive weight updates:

• Outcome tracking: each episode yields an outcome vector of linear/angular tracking rewards, a binary fall indicator, and normalized episode duration, from which a success label and an EMA-based progress signal ($p = r - \bar{r}_i$) are computed per bin.
• History retrieval: a buffer of past context–outcome pairs is maintained, and the $d$0 nearest neighbors of a candidate context are retrieved via KNN, forcing local rather than global reasoning about how rewards evolve with terrain and dynamics.
• Transformer teacher: given retrieved history and the candidate context, a transformer predicts expected reward, success probability, and progress via cross-attention over history tokens. It is trained with a multi-task loss combining MSE reward regression, binary cross-entropy success classification, and weighted MSE progress prediction.

Curriculum expansion proceeds outward from empirically successful bins (those exceeding tracking thresholds), adding their neighbors with asymmetric radii—larger along velocity dimensions for faster expansion toward higher speeds, smaller along terrain and DR axes for gradual robustness growth. Bin weights increase by $d$1, clipped to $d$2, with a fallback score of 1.0 when the history buffer is too small for reliable retrieval. The low-level controller follows the standard privileged-teacher/student distillation setup trained with PPO in Isaac Gym, with no gradients flowing to the curriculum module.

Experimental setup

Training uses 4000 parallel environments at 200 Hz control frequency on a single RTX 4090, completing roughly 400 million steps (~4000 PPO updates) in about four hours. Commands start from $d$3 m/s and expand by $d$4 as tracking improves; starting from wide ranges destabilizes learning. Evaluation uses cost of transport (CoT), a composite stability score over orientation, base height, velocity, joint-limit, self-collision, and torque constraints, and task success rate averaged over trials with 95% confidence intervals.

A bin-resolution ablation across 250, 1000, 4000, and 6000 bins shows coarse binning plateaus around 1.5–2.0 m/s, while 4000 bins reach approximately 6 m/s and 90% of target speed within 8.6M environment steps; 6000 bins marginally improve peak speed but slow early progression slightly, so 4000 is adopted as the operating point.

Results

Simulation speed. At $d$5 m/s, TransCurriculum reaches $d$6 m/s, exceeding RLvRL's 5.5 m/s by approximately 14.55% under comparable command-centric curricula.

Zero-shot hardware transfer. Deployed without fine-tuning on a Unitree Go1, the policy achieves $d$7 m/s on carpet with 90% task success—the highest zero-shot real-world speed among compared methods, surpassing RLvRL's 3.9 m/s on Mini Cheetah by 5.15% and CHRL's 3.45 m/s by 18.8%. Note that these comparisons span different robot platforms, which the authors acknowledge implicitly by reporting per-method transfer robots.

Terrain robustness. Across rigid indoor (carpet 4.1 m/s, 90%; tile 3.1 m/s, 100%), rigid outdoor (cement 3.3 m/s, 80%), deformable (grass 1.8 m/s, 70%), irregular (pebbles 2.1 m/s, 60%; rocks 1.5 m/s, 50%), and sloped surfaces (15–20°, 2.7–3.1 m/s), the policy remains functional without fine-tuning, though speed degrades up to 63% relative to carpet on broken rocks. Disturbance recovery tests show re-stabilization within seconds after bumps and lateral perturbations.

History ablation. The sharpest contrast in the paper is between history-aware and non-history schedulers under identical training conditions:

Metric	Transformer	RNN	MLP
$d$8 at $d$9 (m/s)	6.3	6.1	0.5
CoT	2.60	5.20	—
Stability score	2000	1800	1100
Task success	90%	80%	0%

The MLP scheduler's near-total failure (0.5 m/s, 0% success) indicates that modeling temporal evolution of per-bin rewards is not an incremental improvement but a prerequisite for this curriculum formulation; without memory, the scheduler cannot distinguish productive bins from stagnant ones.

Dimensionality ablation. Comparing command-only, command + DR, and full joint-space curricula reveals a trade-off: the command-only variant attains the fastest simulated speed (6.3 vs. 5.8 m/s for full), but the full curriculum improves stability (1850 → 2000), raises task success (70% → 90%), and reduces sim-to-real transfer loss from 27% to 18% at a commanded 5 m/s—a roughly one-third reduction. The authors' interpretation is that history-awareness primarily enables high-speed training while multidimensionality primarily improves stability and transfer; these effects are complementary rather than redundant.

Limitations and open questions

The paper concedes several boundaries explicitly. Evaluations cover only quadrupeds; no single policy is transferred across morphologies, and bipedal/humanoid locomotion is untested. Real-world failure modes include lateral deviation on straight runs, slips on low-friction and deformable substrates not modeled in simulation, and transient trot-to-crawl gait dropouts (0.5–1.0 s) attributed to contact inference errors. The cross-platform baseline comparison (Go1 vs. Mini Cheetah, A1, ANYmal, custom robots) means reported speed differences conflate curriculum quality with platform capability. Whether the transformer teacher's advantage over RNNs persists at larger bin counts or longer training horizons, and whether the 4000-bin granularity transfers to other robots or task spaces, remain open questions the paper does not resolve.

Conclusion

TransCurriculum demonstrates that moving temporal reward-history modeling from the low-level policy to a transformer-based curriculum scheduler, combined with joint expansion over commands, domain randomization, and terrain, yields measurable gains: 6.3 m/s in simulation, 4.1 m/s zero-shot on hardware, and a reduction of sim-to-real transfer loss from 27% to 18%. The ablations cleanly separate the contributions—history-awareness drives speed, multidimensionality drives stability and transfer—and the MLP ablation underscores that non-history schedulers fail outright in this setting. The approach reduces manual schedule design while remaining compatible with standard PPO-based teacher-student pipelines.