Skill Routing Transformer (SRT)

• Skill Routing Transformer (SRT) is a causal Transformer that routes between transition control and skill execution for complex robotic manipulation.
• It integrates multimodal data from point clouds and robot states to generate discrete stage decisions and temporally extended low-level actions.
• The model leverages a fixed history window and supervised learning on mixed transition and skill datasets to effectively chain learned manipulation skills.

Searching arXiv for the specified paper and closely related context. Skill Routing Transformer (SRT) is the policy module introduced within LodeStar for chaining learned manipulation skills in long-horizon dexterous manipulation tasks (Wan et al., 24 Aug 2025). In the described system, SRT is implemented as a causal, autoregressive Transformer, minGPT-style, that reads a short history of past observations and emits both a discrete “stage” prediction, $$k_t \in \{\text{transition}, \text{skill}_1,\ldots,\text{skill}_K\}$$, and a chunk of low-level actions. Its function is not merely action prediction: it performs routing between a transition controller and a library of pre-trained skill policies, thereby providing the mechanism by which sim-augmented skill learning is composed into coherent long-horizon execution.

1. System role and control semantics

Within LodeStar, SRT is the policy used to chain the learned skills together after demonstrations have been decomposed into semantically meaningful skills and synthetic demonstration datasets have been generated from a few human demos through reinforcement learning (Wan et al., 24 Aug 2025). The policy operates over a mixed dataset of “transition” trajectories and skill-segment trajectories. Each training element has the form $(o_t, a_t^\star, k_t^\star)$, which directly couples observation, action supervision, and stage supervision.

The discrete output $k_t$ has a specific operational meaning. One branch corresponds to “transition,” and the remaining branches correspond to individual learned skills. This means SRT is a hybrid controller: it both decides when to switch and what subsystem should produce the next low-level action. A common misunderstanding is to treat it as a monolithic end-to-end low-level policy. In the described implementation, that interpretation is incomplete. When $k_t=\text{skill}_i$, SRT indexes a skill library and invokes the corresponding pre-trained policy $\pi_{\sigma_i}$; only when $k_t=\text{transition}$ does the SRT action head directly provide low-level Cartesian/joint-space actions. This separation is central to its role in long-horizon manipulation.

2. Observation encoding and Transformer architecture

SRT is implemented as a 4-layer causal Transformer with $L=4$ self-attention blocks, hidden dimension $d_{\text{model}}=256$, 8 attention heads, and a sequence length, or history window, of $H=10$ frames (Wan et al., 24 Aug 2025). The architecture consumes two observation modalities at each timestep: a point-cloud observation and robot state. The point cloud consists of 1024 points, voxelized at 5 mm, and processed by a 3D sparse-conv encoder to produce a per-timestep feature of size $d_p=512$. The robot state, comprising joint positions and velocities, is processed by an MLP to size $(o_t, a_t^\star, k_t^\star)$0.

These modality-specific representations are projected into the common model dimension and summed with learned positional embeddings. The per-timestep token is

$(o_t, a_t^\star, k_t^\star)$1

where $(o_t, a_t^\star, k_t^\star)$2 and $(o_t, a_t^\star, k_t^\star)$3 are the point-cloud and state encoders and $(o_t, a_t^\star, k_t^\star)$4 are learned projections to 256. The sequence

$(o_t, a_t^\star, k_t^\star)$5

is then fed into the 4-layer Transformer.

This formulation makes the tokenization scheme explicitly multimodal while preserving a single causal sequence model. A plausible implication is that the design uses temporal context and cross-modal fusion before routing, rather than deferring modality combination to separate downstream heads.

3. Routing mechanism and action emission

Let $(o_t, a_t^\star, k_t^\star)$6 denote the last-layer Transformer output at step $(o_t, a_t^\star, k_t^\star)$7. SRT computes routing logits

$(o_t, a_t^\star, k_t^\star)$8

with $(o_t, a_t^\star, k_t^\star)$9, and converts them into probabilities by softmax: $k_t$0 This is the stage-classification component of the model and yields the discrete routing decision over transition and skill labels (Wan et al., 24 Aug 2025).

In parallel, an MLP action head $k_t$1 maps $k_t$2 to

$k_t$3

where $k_t$4 is the action-chunk length. During execution only the first of the chunk is used, and the remainder are discarded. The model therefore predicts temporally extended action structure but deploys it in a receding-horizon manner at inference time.

Because the Transformer runs over a window of $k_t$5 past tokens, all routing decisions and action predictions at time $k_t$6 depend on up to $k_t$7 observations, including both point clouds and robot states. This suggests that stage selection is conditioned on short-horizon temporal evidence rather than on a single frame.

4. Supervision, losses, and optimization

SRT is trained by behavior cloning on the mixed dataset of transition trajectories and skill-segment trajectories (Wan et al., 24 Aug 2025). The training objective has two components: a routing loss for stage classification and an action regression loss for low-level action prediction.

The routing loss is

$k_t$8

The action regression loss is mean-squared error: $k_t$9

The total loss is

$k_t=\text{skill}_i$0

with $k_t=\text{skill}_i$1 in the reported experiments. In the provided pseudocode, action supervision is applied to the first $k_t=\text{skill}_i$2 action dimensions of the chunked prediction,

$k_t=\text{skill}_i$3

Optimization uses Adam with learning rate $k_t=\text{skill}_i$4 and batch size 64. A training detail emphasized in the description is teacher-forcing: during training, the model always receives the ground-truth history of observations and does not roll out its own predictions. The mixed dataset includes 1 000 simulated transitions per skill boundary together with real and simulated trajectories for skill segments. In combination, these choices define SRT as a supervised routing-and-action model rather than a policy optimized directly by reinforcement learning.

5. Inference-time execution and integration with the skill library

At inference time, SRT maintains a history buffer of size $k_t=\text{skill}_i$5. For each timestep $k_t=\text{skill}_i$6, the system observes point cloud and state, appends the observation to history, encodes the resulting token sequence through the 3D encoder, MLP, and positional embeddings, and extracts the final Transformer state $k_t=\text{skill}_i$7 (Wan et al., 24 Aug 2025). Routing is then performed by

$k_t=\text{skill}_i$8

The next action depends on the routed stage. If $k_t=\text{skill}_i$9, the action is taken from the SRT action head: $\pi_{\sigma_i}$0 Otherwise, SRT invokes the skill policy indexed by the predicted stage: $\pi_{\sigma_i}$1 The action is then sent to the robot.

The skill library itself is constructed separately. At train time, separate datasets are collected for each skill $\pi_{\sigma_i}$2, and each skill policy $\pi_{\sigma_i}$3 is trained via diffusion-based BC+residual-RL and stored in a library. SRT’s discrete output therefore functions as an interface between sequence-level stage prediction and low-level skill execution. A common misconception is that the Transformer directly parameterizes every contact-rich manipulation behavior. In the stated system, contact-rich manipulation is delegated to the skill policies, while SRT provides switching logic and transition control.

6. Scope, significance, and interpretive boundaries

The described SRT policy successfully learns when to switch between “transition” and the appropriate sub-skill controller, and issues either low-level Cartesian/joint-space actions during transitions or delegates to a learned skill policy $\pi_{\sigma_i}$4 for contact-rich manipulation (Wan et al., 24 Aug 2025). Within LodeStar, this role is tied to the broader claim that sim-augmented datasets enable robust skill training and that SRT effectively chains the learned skills together to execute complex long-horizon manipulation tasks.

Several architectural choices delimit its scope. First, the model’s temporal context is fixed to $\pi_{\sigma_i}$5 frames; consequently, all decisions are made from a short observation history rather than from an unbounded memory. Second, the action head predicts chunks of length $\pi_{\sigma_i}$6, but only the first action is used during execution. Third, stage supervision is explicit: SRT does not discover its routing labels online at deployment, but is trained against ground-truth stage labels $\pi_{\sigma_i}$7 in the mixed dataset.

These details clarify what SRT is and is not. It is a causal autoregressive Transformer for multimodal observation encoding, discrete stage routing, and transition-action prediction. It is not, in the provided formulation, the mechanism that learns each individual skill policy from scratch, nor is it an unrestricted planner over arbitrary horizons. A plausible implication is that its effectiveness depends on the quality of the upstream task decomposition, the fidelity of the skill library, and the coverage of the mixed transition-and-skill dataset that supplies its supervision.