Continuous-Control Benchmarks: Box2D, MuJoCo & PyBullet

• The topic defines continuous-control benchmarks as simulated tasks that test reinforcement learning algorithms in continuous state and action spaces using Box2D, MuJoCo, and PyBullet.
• These benchmarks employ high-fidelity physics engines, standardized task designs, and structured reward mechanisms to assess performance, stability, and generalization.
• Empirical studies and cross-engine comparisons reveal actionable insights into sample efficiency, algorithmic performance, and the simulation-transfer gap.

Continuous-control benchmarks, most notably those based on Box2D, MuJoCo, and PyBullet environments, serve as the primary empirical testbeds for contemporary reinforcement learning (RL) research on continuous action spaces. These benchmarks enable rigorous comparisons of algorithms and models, facilitate reproducibility, and provide insight into the interplay between algorithmic advances and complex, physics-based dynamics. The following reviews the foundational characteristics, design methodology, task details, cross-engine concerns, evaluation protocols, and empirical findings associated with these benchmark suites.

1. Definition and Scope of Continuous-Control Benchmarks

Continuous-control benchmarks are collections of simulated control tasks where agents act in environments characterized by continuous state and action spaces. The dynamics are typically governed by high-fidelity physics engines (MuJoCo, PyBullet, or Box2D), which enable accurate simulation of articulated-body systems such as bipeds, manipulators, and vehicles. The purpose is to assess the performance, stability, and generalization capabilities of RL algorithms—especially policy-gradient, actor-critic, and value-based methods—in settings closer to real-world robotic actuation than discrete-action domains.

The key suites include:

• Box2D-based tasks: e.g., BipedalWalker-v3, CarRacing-v0, LunarLanderContinuous-v2.
• MuJoCo-based tasks: e.g., Hopper-v2, Walker2d-v2, Ant-v2, HalfCheetah-v2, InvertedPendulum-v2, InvertedDoublePendulum-v2, and various morphology variants.
• PyBullet-based tasks: These mirror the MuJoCo suite but are open-source and designed for broader accessibility: AntBulletEnv-v0, HalfCheetahBulletEnv-v0, HopperBulletEnv-v0, Walker2DBulletEnv-v0, HumanoidBulletEnv-v0.

These environments instantiate tasks where the agent must control an articulated system to achieve locomotion, balancing, or manipulation objectives, under possibly stochastic initializations and reward specifications (Ahmed et al., 20 Nov 2025, Xu et al., 2020, Tassa et al., 2020, Mohammed et al., 2020).

2. Environment Architecture and Task Properties

Each environment is specified by its state representation $s_t$, action vector $a_t$, reward structure $r_t$, and termination conditions:

• State spaces: Typically a concatenation of joint positions, velocities, optional contact indicators, and environment-specific features (e.g., lidar readings for BipedalWalker-v3). State dimensionality varies widely, from 4 in InvertedPendulum-v2 up to 376 in HumanoidBulletEnv-v0 (Ahmed et al., 20 Nov 2025, Xu et al., 2020).
• Action spaces: Continuous and bounded, matching the system’s actuation complexity (e.g., 1 in InvertedPendulum, 6 in HalfCheetah, 17 in Humanoid tasks). Actions are element-wise clipped to $[\,-1,\,1\,]$ after normalization.
• Reward functions: Task-dependent but standardized per suite. MuJoCo/PyBullet typically uses $$r_t = v_x - \alpha \lVert a_t \rVert_2^2 + b_{\text{alive}}$$, where $v_x$ is forward velocity and $\alpha$ penalizes action amplitude. Box2D rewards are bespoke but generally integrate progress metrics minus energy/contact penalties (Xu et al., 2020, Ahmed et al., 20 Nov 2025).
• Episode termination: Fixed horizon $T_{\max}$ (most tasks: 1000 steps), early termination on specified events (e.g., fall or crash), or both.

For multi-task RL, differing observation/action vector lengths are aligned by zero-padding to the maximum dimension across all active tasks before inputting to shared networks (Xu et al., 2020).

Table: Selected Continuous-Control Benchmarks

Environment	State Dim	Action Dim	Engine	Task Focus
BipedalWalker-v3	24	4	Box2D	2-D bipedal locomotion
Ant-v2 / AntBulletEnv	111	8	MuJoCo/PyBullet	4-legged locomotion
HalfCheetah-v2 / Bullet	17	6	MuJoCo/PyBullet	Biped running
Hopper-v2 / Bullet	11	3 (MoJ) / 4 (DMC)	MuJoCo/PyBullet	1-legged hopping
Humanoid-v2 / Bullet	376	17	MuJoCo/PyBullet	High-DoF walking

All cited benchmarks use open-source task definitions matching these dimensions; see (Ahmed et al., 20 Nov 2025, Xu et al., 2020, Rahul et al., 2023, Tassa et al., 2020).

3. Reward Structures and Evaluation Metrics

Rewards are environment-specific but constructed to ensure interpretability and comparability:

• MuJoCo/PyBullet locomotion: Rewards consist of progression (forward velocity), action penalties ($\alpha \lVert a \rVert^2$), and possibly alive bonuses. For example, in HalfCheetah-v2, $$r_t = \alpha_{\text{fwd}} v_x - \alpha_{\text{ctrl}} \lVert a \rVert^2$$.
• Box2D tasks: e.g., BipedalWalker-v3 calculates $$r_t = (x_{t+1} - x_t)/\Delta t - 0.0005 \lVert a_t \rVert^2$$, with large penalties for falling and bonus for goal completion (Ahmed et al., 20 Nov 2025).
• DeepMind Control Suite: Rewards bounded to $[0,1]$, with task-specific "tolerance" functions for sparse/dense signals (Tassa et al., 2020).

Evaluation metrics:

• Episode return $G = \sum_{t=0}^{T_{\max}} r_t$ sums the un-discounted or discounted reward.
• Mean/Max return: Averaged across $N_{\text{eval}}$ episodes (typically $10$ in (Xu et al., 2020)).
• Sample complexity: Reported as time-to-threshold or learning curves (mean return vs. training steps).
• Variance: Often plotted as shaded areas or reported as $\pm$ standard deviation over multiple seeds (Ahmed et al., 20 Nov 2025, Tassa et al., 2020).

4. Algorithmic Benchmarks: RL Methods and Hyperparameters

The predominant RL methods evaluated on these suites are on-policy approaches (PPO, TRPO), off-policy deterministic policy gradients (DDPG, TD3), and entropy-regularized actor-critic (SAC), along with distributional variants:

• Network architectures: Actor-critic multilayer perceptrons (MLP) are standard. Common settings include two hidden layers of 256–400 units with ReLU or tanh. Replay buffer sizes are $10^6$ transitions; batch sizes often $256$ (Xu et al., 2020, Grün et al., 2022).
• Optimization: Learning rates $\sim 3\times10^{-4}$–$1\times10^{-3}$, Polyak averaging for target networks ($\tau \approx 0.005$), exploration noise is Gaussian or Ornstein-Uhlenbeck, clipped to keep actions admissible.
• Distributional RL extensions: Recent work extends TD3 and SAC with quantile-based critics (QR, IQN, FQF), showing negligible sensitivity to atom placement or resolution in deterministic continuous-control tasks (Grün et al., 2022).

Hyperparameter normalization across tasks is important; multi-task settings may require uniform input vector handling by zero-padding (Xu et al., 2020).

5. Cross-Engine Comparisons and Generalization

A central concern is the generalization of policy learning across physical simulators:

• Transfer performance: Policies trained on MuJoCo transfer more robustly to PyBullet than the converse. PyBullet-trained agents generally overfit and perform poorly when evaluated in MuJoCo (Mohammed et al., 2020).
• Cross-engine evaluation protocol: For each (algorithm, task, engine), agents are trained to convergence across many random seeds ($\geq$100), then evaluated on alternate-engine environments, with results summarized as mean returns and standard deviations.
• “Simulation–simulation gap”: Transfer performances exhibit high variance, even on solved tasks, necessitating large-scale, multi-seed evaluation for reproducible claims.
• Practical implication: For research aimed at real-to-sim or sim-to-real transfer, MuJoCo remains the preferred engine for development. When open-source requirements mandate use of PyBullet, practitioners must account for the risk of engine-specific overfitting (Mohammed et al., 2020).

Table: Generalization Gaps (Mean Episode Return, HalfCheetah)

Engine (Train → Test)	MuJoCo	PyBullet
MuJoCo → MuJoCo	–1004	–879
MuJoCo → PyBullet	–879	–1004
PyBullet → PyBullet	–486	–1794
PyBullet → MuJoCo	–1794	–486

Values are illustrative, with high per-seed variance (Mohammed et al., 2020).

6. Benchmark-Specific Evaluations and Empirical Trends

Empirical findings and hyperparameter recommendations are environment- and algorithm-dependent:

• Variance and sample efficiency: Policy gradient methods exhibit high variance in return; reward profiling techniques can reduce return variance by up to 1.75× and accelerate convergence to near-optimal return by up to 1.5× (Ahmed et al., 20 Nov 2025).
• Algorithmic performance: DDPG and its successors (TD3, SAC) substantially outperform coarse discretization approaches in continuous control. PPO and TRPO provide competitive on-policy baselines, with faster learning in certain tasks (Rahul et al., 2023, Ahmed et al., 20 Nov 2025).
• Multi-task learning: Methods such as KTM-DRL, employing knowledge transfer from task-specific teachers to multi-task agents, are empirically validated on MuJoCo benchmarks, using zero-padded input handling and hybrid offline-online learning (Xu et al., 2020).
• Distributional RL variants: In PyBullet benchmarks (AntBulletEnv, HopperBulletEnv, HumanoidBulletEnv), the choice of quantile atomization strategy yields no significant difference in final control performance, suggesting the architecture can be selected by computational expedience (Grün et al., 2022).

7. Practical Usage, Extensions, and Community Guidance

• API Conventions: Environments expose standardized Python APIs (reset, step, action_spec, observation_spec), and facilitate both low-level observation and high-level task manipulation (Tassa et al., 2020).
• Pixel-based settings: Some environments support pixel observations and reward-visualization wrappers for vision-based RL.
• Extensibility: Benchmarks support procedural complexity, easy/hard variants, domain randomization, and user-defined task registration.
• Reproducibility and reporting: Reporting mean and variance across many seeds is mandatory for credible evaluation, given the observed stochasticity in train/test splits and engine transfer (Mohammed et al., 2020).

This comprehensive ecosystem of continuous-control benchmarks provides a critical infrastructure for RL research, enabling robust, high-resolution algorithmic comparisons and supporting methodological advances in both single-task and multi-task continuous control domains (Xu et al., 2020, Rahul et al., 2023, Grün et al., 2022, Tassa et al., 2020, Ahmed et al., 20 Nov 2025).