LeRobot v2.1: Open-Source Robot Learning Library

• LeRobot v2.1 is an open-source robot learning library that unifies hardware control, multimodal data management, and asynchronous inference for real-world robotic applications.
• It employs dynamic horizon scheduling, built-in quantization, and real-time safety hooks to optimize inference latency and enhance control across diverse hardware.
• The platform offers a standardized dataset format with hierarchical annotations and extensible middleware integrations, accelerating reproducible research and scalable deployments.

LeRobot v2.1 is a comprehensive open-source library for end-to-end robot learning, designed to unify the robot learning stack from low-level middleware and real-time control to multimodal dataset handling, asynchronous inference, and deployment of state-of-the-art algorithms. Emphasizing accessibility, extensibility, and scalability, it provides an integrated platform for real-world robotics research and large-scale dataset-driven learning, while supporting advanced features for hardware interfacing, data streaming, inference latency optimization, and hierarchical annotation of complex robotic behaviors. The format and library are widely adopted, including as the data backbone for leading datasets such as AIRoA MoMa (Cadene et al., 26 Feb 2026, Takanami et al., 29 Sep 2025).

1. Architectural Overview and Scope

LeRobot v2.1 expands upon its v1.x predecessor by offering a unified stack that encompasses:

• Low-level middleware for real-time communication with a broad range of robot hardware via native ROS 2 (Foxy/Galactic) integration and DDS transport.
• Extensible actuator and sensor plugin models, supporting platforms using EtherCAT-based controllers (e.g., Beckhoff EL5101), FeeTech V3 series, and custom interfaces.
• Data tooling including dataset collection, chunked, network-aware prefetch buffering, prioritized streaming, and multimodal sensor support.
• Optimized asynchronous inference engine with dynamic action horizon scheduling and low-latency model serving.
• Algorithmic support for reinforcement and imitation learning, including RLPD (Reinforcement Learning with Prior Data), HIL-SERL, behavioral cloning with loss mixing, and DiffusionPolicy with variable diffusion steps.

A key distinguishing factor is the elimination of fragmented, hand-tuned subsystems in favor of scalable, data-driven approaches facilitating reproducibility and cross-platform deployment (Cadene et al., 26 Feb 2026).

2. Middleware, Driver Abstractions, and Safety Integration

Middleware Enhancements

LeRobot v2.1 introduces a series of enhancements to its middleware:

• Unified ROS 2 Integration: Native support for ROS 2 nodes and DDS, enabling real-time, low-latency control across multiple hardware platforms.
• ParameterServer Interface: A composable abstraction for dynamic reconfiguration of controller parameters such as joint-PID gains, applicable across heterogeneous robots.
• Plug-in Actuator Drivers: Modules supporting EtherCAT (e.g., Beckhoff EL5101) and FeeTech V3 controllers, with extensibility via subclassing the BaseActuator interface.
• Real-Time Safety Hooks: User-injectable callback mechanisms at 1 kHz control loop frequencies, enabling programmatic assertion of safety invariants or limits.

These middleware extensions support efficient physical deployment and facilitate real-world robotics experimentation by abstracting differences in underlying hardware, communication layers, and control interfaces (Cadene et al., 26 Feb 2026).

3. Data Infrastructure and LeRobot v2.1 Dataset Format

LeRobot v2.1 provides an end-to-end standardized format for multimodal, hierarchical, and synchronized robotic datasets, adopted by large-scale resources such as the AIRoA MoMa dataset (Takanami et al., 29 Sep 2025). Key features include:

• Directory Structure:
  • Root contains metadata.jsonl (episode-wide metadata, one line per episode).
  • Episodes stored in per-ID subfolders with multimodal streams and annotations.
• Per-Modality Schemas (synchronized to 30 Hz):
  • RGB: 480×640, 8-bit PNG images from head and wrist cameras.
  • Proprioception: Joint positions/velocities, head states, etc., stored as float32 arrays in .npz.
  • Force–Torque: Fx, Fy, Fz in N; Mx, My, Mz in N·m.
  • Internals: Base velocities (m/s, rad/s), end-effector pose (position and quaternion orientation).
  • Teleoperation: Raw commands with timing alignment.
• Hierarchical Annotations: Two nested layers—Short-Horizon Tasks (SHT, high-level goals with frame intervals and natural-language descriptions) and ordered Primitive Actions (PAs) that partition the SHT interval, each holding individual success flags.
• Synchronization and Alignment: All modalities are resampled to the common 30 Hz grid; time-alignment tolerates up to ±1/60 s mismatch or flags frame as stale.
• Validation: JSON-Schema and LaTeX formal constraints enforce integrity—e.g., every PA segment strictly and contiguously covers the SHT frames with no overlap, and per-episode metadata must satisfy type and structural constraints.

Table: Key Files and Structures in LeRobot v2.1 Datasets

File/Dir	Content	Synchronization/Constraint
metadata.jsonl	Per-episode summary metadata (JSON lines)	Index for filtering
episodes/epXXXXXX/	Episode data and annotations	Unique ID subdir
rgb/{head,wrist}/	Frame images (480×640 PNG, 30Hz)	Frame-aligned
proprioception.npz	Joint/subsystem states (float32, 30Hz)	timestamps array
ftsensor.npz	Force–torque (float32, 30Hz)	timestamps array
annotations.json	SHT and PA hierarchical labels (JSON)	LaTeX frame constraints

This standardized structure is directly compatible with automated loaders, validation tools, and training pipelines (Takanami et al., 29 Sep 2025).

4. Asynchronous Inference and Scheduling

LeRobot v2.1 implements a generalized asynchronous inference engine supporting adaptive action chunking and on-edge quantization:

• Dynamic Horizon Scheduling: The action chunk length $H_t$ adapts in real time:

  $$H_t = \mathrm{clip}\bigl(H_{\min},\, H_{\max},\, H_{\mathrm{base}} + \kappa\,\delta_t\bigr)$$

  with $\delta_t = q_t - q_{\mathrm{target}}$, where $q_t$ is the current action queue length and $\kappa$ is a smoothing factor.

• Aggregation Function $f_{\alpha,\gamma}$: Merges overlapping action chunks at chunk boundaries to minimize jitter:

  $$f_{\alpha,\gamma}(a^{(1)}, a^{(2)})_k = \alpha a^{(1)}_k + (1-\alpha) a^{(2)}_{k+\gamma H_2}$$

  where $\alpha, \gamma \in [0,1]$.

• Built-in Quantization Hooks: Automatic FP16 and INT8 post-training quantization via le_robot.quantize() enables model deployment with reduced memory and latency on edge hardware.

Empirical Benchmarks

• Peak Memory Reduction: For large policies (π₀, 3.5B params), INT8 quantization achieves a 72% reduction vs. FP32 in v1.5.
• Inference Latency: Average latency for SmolVLA policy decreases from $99.2\pm1.2$ ms (v1.5 FP32) to $35.4\pm0.5$ ms (v2.1 INT8) on RTX 4090.
• End-to-End Cycle Time: On HOPE-JR pick-and-place, async inference (v2.1) reduces episode time from $$H_t = \mathrm{clip}\bigl(H_{\min},\, H_{\max},\, H_{\mathrm{base}} + \kappa\,\delta_t\bigr)$$0 s (v1.5 sync) to $$H_t = \mathrm{clip}\bigl(H_{\min},\, H_{\max},\, H_{\mathrm{base}} + \kappa\,\delta_t\bigr)$$1 s and raises average cubes placed per episode from $$H_t = \mathrm{clip}\bigl(H_{\min},\, H_{\max},\, H_{\mathrm{base}} + \kappa\,\delta_t\bigr)$$2 to $$H_t = \mathrm{clip}\bigl(H_{\min},\, H_{\max},\, H_{\mathrm{base}} + \kappa\,\delta_t\bigr)$$3.

This suggests significant runtime and throughput gains attributable to dynamic horizon scheduling and quantized deployment (Cadene et al., 26 Feb 2026).

5. Supported Algorithms and Learning Paradigms

LeRobot v2.1’s algorithmic API includes:

• RLPD (Reinforcement Learning with Prior Data):
  • Combines off-policy RL gradients with imitation from static datasets,
  • Loss: $$H_t = \mathrm{clip}\bigl(H_{\min},\, H_{\max},\, H_{\mathrm{base}} + \kappa\,\delta_t\bigr)$$4
  • Update: $$H_t = \mathrm{clip}\bigl(H_{\min},\, H_{\max},\, H_{\mathrm{base}} + \kappa\,\delta_t\bigr)$$5
• HIL-SERL: Incorporated as a first-class policy model.
• DiffusionPolicy: Variable diffusion steps $$H_t = \mathrm{clip}\bigl(H_{\min},\, H_{\max},\, H_{\mathrm{base}} + \kappa\,\delta_t\bigr)$$6 scheduled adaptively using control cycle timing.
• Behavioral Cloning: Library supports loss mixing, multi-modal actions, and aligns with the hierarchical annotation structure of datasets such as AIRoA MoMa.

Algorithmic modules are available as ready-to-use Python APIs with deployment-ready quantization and streaming support (Cadene et al., 26 Feb 2026).

6. Hardware Compatibility and Platform Extensibility

New hardware platforms supported in v2.1 include:

• Universal Robots UR5e/UR10e (via ROS 2),
• Kinova Gen3 (EtherCAT interface),
• Franka Emika Panda (v1.5 firmware compatibility),
• Mobile base: Stretch Explorer 3 (via MobileController).

The extensible driver model is based on subclassing and dynamic discovery for both actuators and sensors (including LiDAR, depth cameras, force–torque sensors), supporting rapid extension to new robots and sensor modalities. Plugins can be registered by adding to lerobot.middleware.drivers or via Python entry points, facilitating distributed or cloud-based deployments (Cadene et al., 26 Feb 2026).

7. Validation, Annotation, and Extensibility of Dataset Format

The LeRobot v2.1 data format formalizes storage and annotation, ensuring machine-readability and extensibility:

• Formal Constraints: Enforced via JSON Schema Draft-07 and LaTeX-expressed logical conditions (e.g. non-overlapping, gapless PA coverage of SHT intervals).
• Schema-Driven Extensibility: New sensor streams or annotation layers (such as mid-level "subgoals") can be appended by adding .npz files with 30 Hz timestamp alignment and updating episode or dataset-level schemas.
• Annotation Protocol: The grammar specifies that an episode consists of one SHT and a contiguous, ordered sequence of PAs, whose union of frame intervals exactly matches the SHT interval.

The adoption of this format by AIRoA MoMa and other large-scale datasets illustrates its utility for hierarchical, contact-rich mobile manipulation research and error analysis (Takanami et al., 29 Sep 2025).

---

LeRobot v2.1 thus constitutes an extensible, robust, and high-performance foundation for real-world robot learning, unifying control, data, inference, and learning algorithms in a standardized framework suited for advanced research and scalable deployment (Cadene et al., 26 Feb 2026, Takanami et al., 29 Sep 2025).