Robot Trajectory Markup Language (RTML)

• RTML is a human-readable YAML schema designed to standardize bimanual robot trajectories by encoding both physical and semantic constraints.
• It enforces key principles such as motion stability, pose consistency, and execution efficiency through global and stage-specific constraints.
• RTML integrates with frameworks like RoboCOIN and CoRobot to automate quality assessment, annotation, and unified handling of diverse robot data.

Robot Trajectory Markup Language (RTML) is a human-readable, configuration-focused YAML schema for specifying physical and semantic constraints on bimanual robot demonstration trajectories, introduced as a foundational element in the RoboCOIN dataset and CoRobot data processing framework. RTML enables automated trajectory quality assessment, annotation, and unified handling of heterogeneous multi-embodiment data, enforcing rigorous standards across diverse tasks and platforms (Wu et al., 21 Nov 2025).

1. Design Principles and Motivation

RTML was developed to address the challenge of heterogeneous motion quality in large-scale, cross-platform bimanual robotic demonstration datasets. During the collection of over 180,000 teleoperated robot trajectories using 15 distinct platforms, significant variability in demonstration quality was observed. The schema aims to encode a uniform, programmable criterion of “good” data, enabling downstream learning systems to benefit from standardized, high-quality input.

Three guiding principles define the core of RTML:

• Motion Stability: Ensures trajectories have smooth, predictable profiles without sudden jerks, encoded via velocity and acceleration statistics.
• Pose Consistency: Demands that end-effector poses remain within task-specific tolerance regions, particularly during critical phases such as grasping and placing.
• Execution Efficiency: Prevents both excessive haste (velocity spikes) and unnecessary prolongation (idle or overly long durations), thus balancing speed and precision.

The language supports both global constraints, which apply to the entire trajectory, and stage-wise constraints mapped to semantic subphases (e.g., approach, grasp, transport, release), enabling context-specific parameterization of permissible motion and pose bounds. A formal evaluator (the “RTML checker”) parses the YAML schema to accept, score, or reject trajectories, facilitating quantitative quality control and pre-screening before policy learning.

2. Schema Specification and Syntax

RTML is specified in YAML for ease of human configuration, but its grammar is fully formalized in Extended Backus–Naur Form (EBNF). The main document structure is:

• Task section: Contains task ID.
• Global constraints: Encompasses parameters such as workspace bounds, maximum velocity, acceleration, and duration, applying across the entire demonstration.
• Stages: Sequence of named subphases, each with a unique ID, matched subtask description, and local (possibly stricter or specialized) constraints.

The EBNF excerpt details the complete hierarchy:

<RTMLDocument> ::= "# RTML V1.0" <TaskSection> <GlobalConstraintsSection> <StagesSection> <TaskSection> ::= "task:" INDENT "id:" <QuotedString> <GlobalConstraintsSection> ::= "global_constraints:" INDENT <VelocityConstraint>? <AccelerationConstraint>? <WorkspaceConstraint>? <TemporalConstraint>? <StagesSection> ::= "stages:" INDENT <Stage> { "-" <Stage> } <Stage> ::= INDENT "id:" <QuotedString> "match_subtask:" <QuotedString> "constraints:" INDENT ... ...

An analogous XML Schema (XSD) supports the formalization of these fields for alternate toolchains.

A concrete YAML segment from the “pull_bowl_storage_bread” task illustrates typical construction:

!!!https://www.emergentmind.com/topics/u-1-dirac-spin-liquid!!!! Variables and fields are directly derived from (Wu et al., 21 Nov 2025).

3. Formal Mathematical Semantics

Let a demonstration trajectory be given as: $$T = \left\{(t_i,\mathbf{p}_i,\mathbf{R}_i,g_i)\right\}_{i=1}^N$$ where $t_i \in \mathbb{R}$ is the timestamp, $\mathbf{p}_i \in \mathbb{R}^3$ is position, $\mathbf{R}_i \in SO(3)$ is the end-effector orientation (as a 3×3 rotation matrix), and $g_i \in [0,1]$ is the gripper state.

Discrete velocities and accelerations are computed via: $$\mathbf{v}_i = \frac{\mathbf{p}_{i+1} - \mathbf{p}_i}{t_{i+1} - t_i}, \quad \mathbf{a}_i = \frac{\mathbf{v}_{i+1} - \mathbf{v}_i}{t_{i+1} - t_i}$$ Statistical constraints are defined by: $$\bar v = \frac{1}{N-1} \sum_{i=1}^{N-1} \| \mathbf{v}_i \|, \quad \sigma_v = \sqrt{ \frac{1}{N-1} \sum_{i=1}^{N-1} ( \| \mathbf{v}_i \| - \bar v )^2 }$$

$T_{\text{dur}} = t_N - t_1$

$$\theta_i = \arccos\left( \frac{ \mathrm{tr} ( \mathbf{R}_i^\top \mathbf{R}_{\rm ref} ) - 1 }{2} \right )$$

Typical global constraints require, for all relevant $i$, e.g.: $$\| \mathbf{v}_i \| \le v_{\text{max}}, \quad \bar{v} \le \texttt{mean\_max}, \quad \sigma_v \le \texttt{std\_max}, \quad T_{\text{dur}} \in [ \texttt{duration\_min},\, \texttt{duration\_max} ]$$ Local (stage-wise) constraints apply the same formulae to the temporal region associated with each semantic subphase, as identified via annotation.

A plausible implication is that the schema is agnostic to whether constraints are applied in Cartesian space (positions, rotations) or joint space, simply by selecting the relevant primitives.

4. Integration into CoRobot Framework

RTML underpins multiple key functions in the CoRobot framework:

• Trajectory Quality Assessment: The RTML checker evaluates each trajectory against global and local constraints. The assessment process entails segmentation of raw data into named “stages” (using timestamps and subtask annotations), rule enforcement, per-phase violation reporting, and the assignment of an overall pass/fail score. Notably, up to 35% of raw demonstrations were filtered for quality control, directly increasing policy robustness (Wu et al., 21 Nov 2025).
• Automated Annotation Generation: RTML’s match_subtask field seeds the annotation pipeline. Rule-based keyframe extraction and LLMs refine subtask labels. Thresholding methods, derived from RTML velocity and pose limits, automatically demarcate phase boundaries (e.g., transitions marked by sustained low velocity are linked to “idle_arm” detection). Frame-level kinematic attributes are generated via sliding-window analysis, referencing RTML constraints.
• Unified Multi-Embodiment Data Handling: RTML schema files are stored alongside every data unit. By explicitly encoding embodiment- and task-specific constraints, disparate robot morphologies (e.g., 6-DoF arms, grippers, dexterous hands) can be subjected to identical pre-processing, annotation, and filtering pipelines. The “atomic storage” system enables researchers to select subsets using RTML metadata such as task IDs, robot types, or scenarios.

5. Typical Constraint Fields and Example Table

The following summarizes common RTML fields, derived from both EBNF and the provided YAML example:

Constraint Type	Field Example	Typical Usage
Global Constraints	velocity.linear.max, acceleration.linear.max	Bounds on overall trajectory kinematics
Workspace Constraints	workspace.right.min/max	Enforces spatial bounds per arm
Temporal Constraints	temporal.duration_min/max	Regulates (sub)task durations
Orientation Constraints	orientation.left.angular_mean_deviation_max	Maintains orientation accuracy
Idle Arm Constraints	idle_arm.velocity_linear_mean_max	Restricts non-dominant arm movement

This explicit field hierarchy supports both human legibility and machine evaluability, enforcing cross-platform consistency.

6. Mathematical and Algorithmic Underpinnings

RTML’s structure reflects explicit mathematical foundations:

• Kinematic and Dynamic Statistics: Motion constraints are imposed on direct functions of observed position and orientation trajectories, with formal requirements for empirical means, variances, and extrema.
• Phase Segmentation: Temporal decomposition leverages velocity and acceleration thresholds; e.g., periods where $\| \mathbf{v}_i \| < v_{\text{idle}}$ over a fixed $\Delta t$ are marked as “idle_arm,” while intervals breaching set thresholds are flagged as anomalies.
• Symbolic Task Binding: The match_subtask string binds each specified phase to discovered intervals $[t_{\text{start}}, t_{\text{end}}]$, computed via rule-based detectors within CoRobot, supporting both programmatic annotation and semantic traceability.

A plausible implication is that the same schema can be extended to joint-space or alternative modalities by substituting the primitive fields, maintaining schema flexibility.

7. Impact and Significance in Multi-Embodiment Learning

RTML provides the unifying schema + rule-set guaranteeing that every collected and released trajectory satisfies a configurable and formally specified standard of both physical and semantic quality. By propagating RTML YAML files alongside data, all downstream processes—quality filtering, annotation, training, and comparative analysis—can be conducted with uniform rigor across embodiments. This enables reliable, reproducible, and scalable bimanual robot learning at the dataset and platform level (Wu et al., 21 Nov 2025).