Continuous-Control Benchmarks

• Continuous-control benchmarks are environments defined by continuous-state and continuous-action MDPs, enabling rigorous evaluation of reinforcement learning and control strategies.
• They standardize state/action encodings, reward normalization, and simulation protocols using engines like MuJoCo and Unity to ensure cross-method comparison.
• These benchmarks drive algorithmic innovation by providing reproducible task suites across domains such as robotics, energy systems, and biomechanical simulations.

Continuous-control benchmarks constitute a foundational class of test environments and task suites used for rigorously evaluating reinforcement learning (RL), optimal control, and related algorithmic paradigms. These benchmarks formalize Markov Decision Processes (MDPs) with continuous, typically bounded, action and state spaces and serve as canonical references for progress, reproducibility, and algorithmic comparison in control-oriented machine learning research. Modern continuous-control benchmarks span simulated musculoskeletal systems, locomotion and manipulation robots, power grid operation, and partially observed high-dimensional systems. Benchmark suites such as the DeepMind Control Suite, D4RL, RLBench, MuJoCo locomotion tasks, and domain-specific environments (e.g., RobocupGym, OPF with battery storage) define precise state/action encodings, reward structures, constraints, and evaluation standards, often with open-source implementations, that enable rigorous cross-methodology comparison (Tassa et al., 2018, Ding et al., 2023, Seyde et al., 2024, Rajeswaran et al., 2017, Beukman et al., 2024, Liu et al., 1 Feb 2025).

1. Design Principles and Structure of Continuous-Control Benchmarks

Continuous-control benchmarks enforce a standardized problem structure—typically continuous-state, continuous-action MDPs with formalized interfaces, reward normalizations, and dynamics—intending to foster reproducibility and well-calibrated evaluation (Tassa et al., 2018, Duan et al., 2016). States commonly encode system configuration in ℝⁿ, including joint angles/velocities, extrinsic features, and often auxiliary sensors (contacts, IMUs), while actions are low-dimensional vectors a ∈ [–1, 1]^d, representing torques, forces, or control velocities. Physics simulation is typically driven by engines such as MuJoCo (DeepMind Control Suite (Tassa et al., 2018)), Unity PhysX (Marathon Environments (Booth et al., 2019)), Box2D, proprietary simulators (e.g. rcssserver3d in RobocupGym (Beukman et al., 2024)), or custom domain-specific solvers (e.g. OPF power grid (Ding et al., 2023)).

Reward design is interpretable and normalized: DeepMind Control Suite, for example, constrains r(s, a) ∈ [0, 1] for dense tasks and {0, 1} for sparse rewards, often employing composite or tolerance-based mapping to enable uniform scale across tasks (Tassa et al., 2018). Episode lengths are typically fixed (e.g., 1,000 steps), with episodic returns accordingly bounded (e.g., [0, 1,000]), facilitating direct comparison of learning curves and aggregate performance. MuJoCo-based domains leverage physics models defined in MJCF and use semi-implicit Euler or higher-order integrators to avoid numerical instabilities.

Environmental diversity is achieved through taxonomy over task classes: simple pendulums, planar and articulated manipulation, legged locomotion, fluid/swimmer tasks, and high-dimensional humanoids (states ranging from dim=3 up to 376, action dimensionality up to 56 in Humanoid_CMU (Tassa et al., 2018)). Recent benchmarks have extended this by introducing hard constraints (equality/inequality) (Ding et al., 2023), vision-based observation modalities (Wang et al., 2024, Scannell et al., 1 Mar 2025), and partial observability (Beukman et al., 2024, Duan et al., 2016).

2. Benchmark Suites and Canonical Tasks

The landscape of continuous-control benchmarks is anchored by several widely adopted suites and specialized environments:

Suite/Domain	Key Tasks / Domains	State Dim	Action Dim	Reward Norm	Unique Features
DeepMind Control Suite	Pendulum, CartPole, Reacher, Finger, Ball-in-cup, Manipulator, Hopper, Cheetah, Walker, Humanoid, Humanoid_CMU	2–124	1–56	[0,1]	Standardized rewards, MuJoCo, MJCF
OpenAI/MuJoCo Gym	Hopper, Walker2d, HalfCheetah, Ant, Humanoid	11–376	2–17	Task-specific	Legacy for RL, robust baseline
RLBench	20+ manipulation tasks (vision+proprio)	~50–100	up to 20	Success rate or custom	Sparse reward, multi-view, demos
D4RL	MuJoCo locomotion (offline datasets), adroit, antmaze, manipulation	mul.	mul.	[0,100]	Offline+online data, normalized ret.
Marathon/Unity	Hopper, Walker, Humanoid, Ant	31–88	4–21	Custom	Unity ML-Agents, multi-agent
RobocupGym	SimpleKick, VelocityKick (Nao soccer humanoid)	56	20–22	Task-specific	rcssserver3d; contact, 22-DOF, partial obs.
Hard-constraint tasks	Safe-CartPole, Spring Pendulum, OPF w/Battery	6–57	2–43	Task-specific	Explicit constraints, equality + ineq.

Each benchmark formalizes the critical challenge settings—state encoding, action bounds, reward shaping and sparsity, termination logic, and optional episode randomization or perturbations—for scientific rigor and fair algorithmic comparison (Tassa et al., 2018, Rajeswaran et al., 2017).

3. Reward Functions, Constraints, and Evaluation Protocols

Reward specification in continuous-control benchmarks typically operationalizes performance objectives via analytic tolerance functions, Lorentzian/step-wise shaping, or composite normed combinations. In the DeepMind Control Suite, the key reward utility is:

• $$r(s,a) = \text{tolerance}(x; (\ell, u), \text{margin}, \text{sigmoid}) \in [0,1]$$

where $x$ is any scalar task-relevant feature, and the utility saturates inside bounds and decays smoothly outside, supporting either multiplicative or weighted composition of sub-objectives (Tassa et al., 2018). Locomotion tasks use forward speed (e.g., $r(v) = \max(0, \min(v/10, 1))$ for Cheetah), sparse benchmarks enforce region-based indicators, and swimmer/fish employ Lorentzian metrics.

Recent benchmarks emphasize hard-constraint enforcement. For example, "Safe CartPole" requires satisfaction of both an equality constraint (zero net vertical force) and an inequality constraint (bounded horizontal force), formalized as:

• Equality: $f_1\sin\theta_1 + f_2\sin\theta_2 = 0$
• Inequality: $f_-\leq f_1\cos\theta_1 + f_2\cos\theta_2 \leq f_+$

Constraint violations are penalized or terminate the episode, and reported as maximum instantaneous and episodic infraction metrics (Ding et al., 2023).

Evaluation is protocol-driven: per-episode return is summed over a fixed horizon (commonly 1,000 steps), with mean±SE computed over fixed seeds and standardized reporting of learning curves both by environment steps and, increasingly, wall-clock time. Best practices specify per-task hyperparameter constancy, sufficient seed averaging (e.g., ≥5 seeds), and alignment to benchmark-dictated reward normalization, facilitating reliable algorithmic sorting (Tassa et al., 2018).

4. Algorithmic Baselines and Comparative Results

Continuous-control benchmarks have catalyzed the proliferation of deep RL algorithms, enabling comparative performance analysis under controlled conditions. Baselines span actor-critic, model-based, value-based, policy-gradient, and hybrid approaches, including A3C, DDPG, D4PG, TD3, SAC, PPO, Dreamer, actor-critic neuroevolution, auto-regressive value decomposition, world-model predictive control, and search-based and optimization-based planners (Tassa et al., 2018, Franke et al., 2019, Liu et al., 1 Feb 2025, Scannell et al., 1 Mar 2025).

Empirical findings consistently underscore the data efficiency and asymptotic return hierarchy across methods:

• D4PG > DDPG > A3C on DeepMind Control Suite (Tassa et al., 2018)
• Adaptive and auto-regressive value discretization (e.g. ARSQ, GQN) often matches or exceeds state-of-the-art baselines, especially in sparse-reward or high-dimensional settings (Seyde et al., 2024, Liu et al., 1 Feb 2025)
• Bang-bang/binary-discrete policy heads can match Gaussian-continuous policies for rewards, particularly when action penalties are absent or minimal (Seyde et al., 2021). However, with strong action cost or smoothness constraints, discrete and continuous baselines exhibit differentiated trajectories (Seyde et al., 2024).
• Constrained RL benchmarks demonstrate that methods tailored to explicit equality/inequality constraints (e.g. Reduced Policy Optimization) robustly surpass Lagrangian-penalty or unconstrained variants in both return and constraint satisfaction (Ding et al., 2023).

Aggregate learning curves and tabulated mean returns (see Table 1 below) are the standard output; for example:

Task	A3C	DDPG	D4PG
cartpole:swingup	780 ± 25	910 ± 15	980 ± 5
cheetah:run	410 ± 40	670 ± 30	890 ± 10
manipulator:bring	120 ± 35	300 ± 25	650 ± 20

5. Benchmarking Challenges, Realism, and Extensions

Empirical analysis and theoretical insight reveal challenges in continuous-control benchmarking:

• Bang–bang action bias: Absence of energy or smoothness penalties in rewards leads many algorithms to extremal (boundary-value) actions, confounding evaluations of policy robustness, exploration, and real-world transfer (Seyde et al., 2021, Seyde et al., 2024).
• Distribution shift effects: High-dimensional action spaces (e.g. Humanoid) exacerbate distorted action density under tanh-squashed Gaussians (as in SAC), resulting in mode misalignment and suboptimal policies unless the induced Jacobian is corrected (Chen et al., 2024). Benchmarks such as "HumanoidBench" were introduced to expose these high-d phenomena.
• Reward shaping and termination: Narrow initial state distributions and early termination can induce brittle, trajectory-centric solutions. Diverse initialization and omission of premature resets yield policies that generalize to perturbations and recover from falls (Rajeswaran et al., 2017).
• Partial observability and real-world applicability: RobocupGym and power-grid benchmarks incorporate partial observation, rich contact dynamics, constraints, and exogenous disturbances, demanding memory-augmented or robust control policies (Beukman et al., 2024, Ding et al., 2023).
• Algorithmic bottlenecks: Off-policy methods (notably SAC) may be sample-inneficient or unstable under large-scale parallelization, particularly in CPU-bound or communication-heavy simulation settings (Beukman et al., 2024).

Benchmark extensions emphasize (1) vision-based and high-dimensional input modalities (Scannell et al., 1 Mar 2025, Wang et al., 2024), (2) hard-constraint and safety-critical control (Ding et al., 2023), (3) hierarchical/long-horizon challenges (Duan et al., 2016), (4) multitask and transfer protocols, and (5) partial observability and adversarial environments.

6. Usage, Customization, and Best Practices

Benchmarks are designed for extensibility and open experimentation. For the DeepMind Control Suite:

• Environments are loaded and stepped in Python with dm_control.suite; action and observation specs are programmatically queried.
• Pixel-only observations and low-level MuJoCo physics access are available via wrappers and direct simulation interfaces.
• Custom tasks and domains may be created by modifying MJCF XMLs, Python reward shaping, or by injecting new task entries in domain registries.
• Rigorous experimental reporting mandates: fixed episode length, mean±SE across ≥5 seeds, publication of hyperparameters and random seeds, and standardized reward normalization (Tassa et al., 2018).

RobocupGym natively supports vectorized training over distributed TCP-connected simulators and integrates seamlessly with Stable Baselines 3. Unity Marathon Environments leverage batch agent design and CPU-based simulation for high-throughput, multi-agent learning (Booth et al., 2019, Beukman et al., 2024).

Researchers are advised to:

• Benchmark new algorithms against simple linear/RBF policy baselines (Rajeswaran et al., 2017) and discrete as well as continuous policy heads (Seyde et al., 2021), to calibrate claimed improvements.
• Rigorously report the method of deterministic evaluation (e.g., tanh(μ) vs. true mode correction), especially in high-d settings (Chen et al., 2024).
• Design experimental protocols to reveal not only asymptotic return, but also sample efficiency, seed variance, constraint satisfaction, and policy robustness to perturbations and environmental variation.

7. Impact, Limitations, and Future Directions

Continuous-control benchmarks have been instrumental in standardizing benchmarks for RL and control research, enabling reproducibility and scientific rigor, and have motivated algorithmic innovation across actor–critic, model-based, and constraint-satisfying methodologies. However, limitations remain:

• Many canonical benchmarks remain easier than real-world control in terms of dynamics complexity, observation noise, and robustness demands (Seyde et al., 2021).
• Dense reward structures and hand-crafted terminations may artificially simplify problem difficulty and encourage solutions with little transfer value.
• Algorithmic progress may reflect better exploitation of benchmark-specific design idiosyncrasies (e.g., reward normalization, action-bound enforcement) rather than fundamental advances.

Continued development points towards richer task spaces (curriculum-based, hierarchical, and multi-agent settings), the incorporation of domain randomization, sim-to-real protocols, rigorous partial observability, and adoption of robust and constraint-aware evaluation metrics. Emergent benchmark suites integrating hard constraints, hybrid action modalities, and real-world-inspired tasks (e.g., smart grids, soccer robotics) point toward a trend of increasing realism, with the aim of achieving not only high return but robust, scalable, and generalizable policies (Ding et al., 2023, Beukman et al., 2024, Scannell et al., 1 Mar 2025).