D4RL Adroit Benchmarks

• D4RL Adroit Benchmarks are a suite of high-dimensional dexterous manipulation tasks in MuJoCo using the 24-DoF ShadowHand, targeting offline reinforcement learning.
• They cover tasks like pen twirling, hammer striking, door opening, and object relocation with varied datasets (human, expert, cloned) to assess algorithm performance.
• The benchmarks expose challenges such as sparse rewards, non-Markovian human demonstrations, and extreme action-space dimensionality, driving innovations in conservative value estimation and policy alignment.

The D4RL Adroit Benchmarks are a canonical suite of high-dimensional dexterous manipulation tasks in MuJoCo, designed to probe the limits of offline reinforcement learning algorithms. Featuring the 24 DoF ShadowHand and spanning fine-motor tasks such as pen spinning, hammer manipulation, door opening, and object relocation, these benchmarks expose unique statistical and algorithmic challenges posed by narrow, multimodal human data, brittle reward landscapes, and extreme action-space dimensionality. Their adoption has catalyzed algorithmic advances in conservative value estimation, support-aware regularization, policy alignment, and reward-aligned human-like control.

1. Task Structure, State-Action Spaces, and Dataset Construction

The Adroit suite comprises four manipulation tasks (Fu et al., 2020):

• Pen: Continuous twirling of a free pen.
• Hammer: Striking a fixed nail using a grasped hammer.
• Door: Grasping and pulling open a door handle to a target angle.
• Relocate: Picking up and placing a ball at a specified target.

All use MuJoCo’s 24-DoF ShadowHand, yielding observation vectors of size 78–111 (joints, object state, sensor flags) and action vectors in $\mathbb{R}^{24}$ (target joint torques/velocities).

Each environment features three canonical offline dataset variants:

• Human: Teleoperated demonstration, 5k–11k transitions each.
• Expert: DAPG-finetuned agent, $\sim$500k–1M transitions, near-tightly clustered returns.
• Cloned: 50-50 mixture of BC agent rollouts and original demos, $\sim$500k–1M transitions.

Reward functions combine sparse goal shaping and a per-step action penalty: $$r(s_t, a_t) = -\| \text{target\_object}(s_t) - \text{goal} \|_2 - 10^{-3} \|a_t\|_2^2$$ with specifics for pen spin rate, hammer/nail tip, door handle/hinge, or ball/target position.

2. Algorithmic Challenges and Baseline Performance

Adroit highlights several intrinsic offline RL challenges (Fu et al., 2020):

• High-Dimensional Control: Estimation errors in $Q$-values scale unfavorably with DoF.
• Sparse, Shaped Rewards: Poor credit assignment from delayed, infrequent reward signal.
• Narrow Human Data: Behavioral coverage is low-entropy and highly structured.
• Non-Markovian Human Demos: Trajectories exhibit patterns not well modeled by simple stochastic policies.

Canonical results for key algorithms on human/cloned/expert datasets reveal striking phenomena:

Task	BC	BEAR	BCQ	CQL
pen-human	34.4	-1.0	68.9	37.5
hammer-human	1.5	0.3	0.5	4.4
door-human	0.5	-0.3	-0.0	9.9
relocate-human	0.0	-0.3	-0.1	0.2

Conservative and support-aware algorithms (BCQ, CQL) are dramatically more stable on expert data, but often fail to improve on human-only data beyond simple behavior cloning.

3. One-Step Offline RL: Objective, Implementation, and Adroit Results

The one-step policy improvement paradigm, as formalized in "Offline RL Without Off-Policy Evaluation" (Brandfonbrener et al., 2021), avoids iterative off-policy evaluation and its instability. The approach:

• Fits behavior policy $\hat\beta(a|s)$ via maximum likelihood (BC).
• Estimates on-policy $Q^{\beta}$ through TD minimization:

$$\min_Q \sum_{(s,a,r,s') \in D} [Q(s,a) - (r + \gamma \mathbb{E}_{a' \sim \hat\beta} Q(s', a')) ]^2$$

• Performs one step of policy improvement against this $Q^{\beta}$ using:
  • Reverse-KL regularization
  • Easy-BCQ (support constraint)
  • Exponentially weighted imitation

Hyperparameters are swept modestly (e.g., Reverse-KL $\alpha \in [0.03, 10]$) and evaluation is direct.

Task	Prior Iterative	BC	Easy BCQ	Rev. KL Reg.	Exp. Weight
pen-c	56.9	49.3	67.0 (best)	55.3	54.7
hammer-c	2.1	0.5	2.8 (best)	0.2	1.2
relocate-c	-0.1	0.0	0.3 (best)	0.1 (best)	0.1 (best)
door-c	0.4	0.0	0.4	0.0	0.1

One-step variants consistently match or surpass prior iterative approaches, particularly where behavior data has favorable state coverage, thus reducing variance in $Q^{\beta}$ and improving robustness to hyperparameter choice.

Analysis identifies two failure modes of iterative/off-policy algorithms avoided by one-step: (i) escalating variance from distribution shift; (ii) iterative error exploitation in underexplored state-action corners (Brandfonbrener et al., 2021).

4. Policy Alignment and Extraction: AlignIQL and Related Methods

AlignIQL introduces a constrained optimization perspective to implicit policy extraction from value functions (He et al., 28 May 2024). Given a learned (Q,V) via expectile regression, the procedure recovers the implicit policy $\pi^*$ by:

$$\min_{\pi} \mathbb{E}_{s, a \sim \pi} [ f(\pi(a|s)/\mu(a|s)) ], \text{ s.t. } \mathbb{E}_{a \sim \pi}[Q(s,a)] = V(s)$$

with $f$ a convex regularizer (typically $\log$), yielding solutions proportional to $\mu(a|s) \exp(|\beta(s)| Q(s,a))$ under appropriate conditions.

Two practical algorithms:

• AlignIQL-hard: Lagrange multipliers enforce exact constraint satisfaction, requiring auxiliary multiplier networks.
• AlignIQL: A soft-constraint variant, reduces to reweighting by $\exp[|\eta| (Q-V)]$ without multipliers.

Experimental results show state-of-the-art normalized returns on Adroit human and expert datasets (e.g., pen-human: 76.0$\pm$4.8, beating BC/BCQ/IQL[CQL]), with significant robustness to hyperparameters and sample counts. The approach leverages a conditional diffusion model for $\mu(a|s)$ to handle multi-modal human data (He et al., 28 May 2024).

5. Model-Based and Latent Action Approaches: TAP, MAQ, and Human-Likeness

Recent directions focus on learning low-dimensional latent action or macro-action spaces for efficient planning and human-like behavior.

Trajectory Autoencoding Planner (TAP) (Jiang et al., 2022):

• Learns a state-conditional VQ-VAE to encode trajectories into discrete latent codes.
• Planning is performed via search over code sequences, optimizing for both predicted return and likelihood under the data distribution.
• Achieves normalized returns of 76.5 (pen-human), 8.8 (door-human), and 127.4 (pen-expert), consistently surpassing trajectory-transformer and model-free actor-critic baselines while maintaining constant decision latency even as action space dimensionality scales.

Macro Action Quantization (MAQ) (Guo et al., 19 Nov 2025):

• Distills human demonstrations into a low-cardinality codebook of H-step macro actions using a conditional VQVAE.
• RL policy and critic operate in this macro-action space, incentivizing reward maximization while restricting policies to human-like temporal segments.
• Evaluation includes behavioral similarity (DTW/WD metrics), success rates, and human-judged Turing and human-likeness ranking tests, with MAQ+RLPD achieving a 39% Turing test "win-rate" and highest trajectory similarity and ranking among evaluated agents.

Both TAP and MAQ frame planning or control as search or selection over latent codebooks learned from data, promoting both support-aware exploration (reducing OOD errors) and, for MAQ, demonstrable stylistic alignment with human demonstrators.

6. Evaluation, Metrics, and Analytical Protocols

D4RL prescribes the following protocol (Fu et al., 2020):

• Train-test splits for hyperparameter selection (e.g., training on pen/door, evaluating on hammer/relocate).
• Normalized score: $$100 \cdot (J(\pi) - J_{random}) / (J_{expert} - J_{random})$$, with $J_{random}$ and $J_{expert}$ from environment rollouts.
• Success rate for goal achievement (where applicable).

Advanced metrics for newer methods include:

• Trajectory similarity: Dynamic Time Warping (state/action), Wasserstein distance.
• Human-likeness: 2AFC Turing tests and pairwise human-likeness ranking.
• Decision latency: Time per action selection (TAP demonstrates $<0.05$ s/action across high-dimensional tasks).

7. Analytical Insights and Open Problems

D4RL Adroit remains a crucible for advances in offline RL, chiefly due to:

• Exacerbation of distributional shift and extrapolation errors in narrow, high-dimensional demonstration domains.
• Necessity for support-aware, conservative, and regularized algorithms that remain tractable and robust.
• Human-likeness as a practical and scientific desideratum for learned behaviors, now systematically measurable via trajectory and Turing-like tests (Guo et al., 19 Nov 2025).
• Latent/action macro-structure learning as a unifying theme in algorithms attaining both statistical efficiency and interpretability.

Current high-performing algorithms (AlignIQL, TAP, MAQ variants, one-step regularized) achieve either state-of-the-art returns, strong stability, or high-fidelity imitation, but full unification of task success, human-likeness, and generalization in the face of narrow or multimodal data distributions continues to drive algorithmic research (Brandfonbrener et al., 2021, He et al., 28 May 2024, Jiang et al., 2022, Guo et al., 19 Nov 2025).