Behavior Transformer (BeT) Overview

• Behavior Transformer (BeT) is a transformer-based imitation-learning method that models continuous, multi-modal actions using k-means discretization paired with continuous offsets.
• It employs a decoder-only transformer with separate categorical and offset heads to capture historical context and enable robust sequence prediction.
• Extensions such as C-BeT and VQ-BeT enhance goal conditioning, scalability, and inference speed, broadening applications in robotics, autonomous driving, and beyond.

Behavior Transformer (BeT) is a transformer-based imitation-learning architecture for modeling unlabeled, multi-modal demonstration data with continuous actions. Introduced to address the mismatch between token-centric transformers and continuous control, BeT factorizes each action into a discrete action bin obtained by $k$-means and a continuous offset, thereby allowing a decoder-only transformer to predict multi-modal continuous actions without reward labels or task segmentation (Shafiullah et al., 2022). Subsequent work extended this template to future-conditioned play data, hierarchical vector quantization, online residual adaptation, and token-based task adaptation, so the term now denotes both the original method and a family of transformer-centric behavior models built around action tokenization and sequential context (Cui et al., 2022).

1. Historical setting and conceptual scope

BeT emerged from the observation that behavioral cloning and Offline RL often assume reward-labeled data or effectively unimodal action distributions, whereas human demonstrations are commonly unlabeled, suboptimal, noisy, and inherently multi-modal (Shafiullah et al., 2022). In robotics and driving, multiple valid actions may solve the same local state, and errors can compound over long horizons. The original BeT formulation targeted precisely this regime: large, pre-collected datasets of demonstrations with wide variance and no reward annotation.

A central conceptual distinction is that the original BeT is unconditional. It models behavior directly from observation history and demonstration statistics, rather than from explicit goal tokens or reward functions. Later variants added conditioning and adaptation mechanisms, but those are extensions rather than properties of the base model. This is important because BeT is sometimes conflated with later goal-conditioned or foundation-model-style systems; in the original formulation, its primary contribution is multi-modal continuous-action prediction via discretization-plus-correction, not generic promptability or universal controllability (Cui et al., 2022).

The problem formulation also differs from standard language modeling in a specific way emphasized by later work: decision making requires modeling actions as continuous-valued vectors whose distributions are multi-modal, may come from uncurated sources, and whose generation errors can compound in sequential prediction (Lee et al., 2024). BeT can be understood as an attempt to import transformer sequence modeling into that setting while preserving a faithful action interface.

2. Core formulation: action bins, offsets, and autoregressive sequence modeling

The canonical BeT pipeline begins by clustering all actions in the dataset into $k$ cluster centers $\{A_1,\dots,A_k\}\subset\mathcal{A}$. Each action $a\in\mathbb{R}^d$ is then decomposed into a discrete bin index and a continuous residual:

$$\lfloor a \rfloor := \arg\min_i \|a - A_i\|_2, \qquad \langle a \rangle := a - A_{\lfloor a \rfloor}, \qquad a = A_{\lfloor a \rfloor} + \langle a \rangle.$$

This factorization makes the action prediction problem compatible with standard transformer heads: a categorical head predicts the action bin, and a continuous head predicts the offset associated with each bin (Shafiullah et al., 2022).

Architecturally, BeT uses a decoder-only transformer, described in the paper as a minGPT variant, to process observation sequences and output a distribution over $k$ discrete action bins per timestep. Because rare modes and class imbalance are central to behavior data, the discrete head uses focal loss rather than plain cross-entropy:

$$\mathcal{L}_{\text{focal}}(p_t)=-(1-p_t)^\gamma \log(p_t).$$

To recover action fidelity lost through quantization, BeT adds a multi-task offset head. Only the offset corresponding to the ground-truth bin is penalized:

$$\text{MT-Loss}\left(\mathbf{a}, (\langle \hat a^{(j)} \rangle)_{j=1}^k\right) = \sum_{j=1}^k \mathbb{I}[\lfloor \mathbf{a} \rfloor = j]\cdot \|\langle \mathbf{a} \rangle - \langle \hat a^{(j)} \rangle\|_2^2.$$

At inference time, the model samples an action bin from the categorical distribution, retrieves the predicted offset for that bin, and reconstructs the action as $$\hat a_t = A_{t,j} + \langle \hat a_t^{(j)} \rangle$$ (Shafiullah et al., 2022).

The resulting model is not merely a discretized BC baseline. Its sequence model explicitly exploits historical context, which the original work identifies as important for non-Markovian dependencies and mode disambiguation. The ablation study reported that discrete binning is critical for enabling multi-modality, action offsets substantially improve performance in high-dimensional action spaces, and transformer backbones outperform MLP, temporal convolution, LSTM, and MDN alternatives in leveraging history and context. Uniform quantization and continuous/unimodal ablations underperform the full $k$-means-plus-offset design (Shafiullah et al., 2022).

3. Conditional behavior generation from play data

Conditional Behavior Transformer (C-BeT) extends BeT from unconditional behavior modeling to future-conditioned control from play data. The extension keeps BeT’s multi-modal action prediction mechanism but augments the input with both current observation sequences and future or goal observation sequences, yielding a conditional policy of the form

$k$0

Training data are constructed by selecting future times $k$1 within play trajectories and treating future observations as goals; the paper describes this as hindsight relabeling, so every future state may serve as a potential goal (Cui et al., 2022).

This extension is motivated by the structure of play data: uncurated, diverse, noisy robot demonstrations collected from non-experts, typically without task labels or rewards. C-BeT uses the same action discretization and offset-correction machinery as BeT, but applies it to pairs of current and future observation sequences. The method therefore turns a sequence model of demonstrations into a controllable policy extractor. In the paper’s framing, the user can supply a goal frame or other future observation at test time, and the model generates behavior directed toward that goal without explicit task labels or reward engineering (Cui et al., 2022).

Empirically, C-BeT establishes that goal conditioning is not a superficial modification. On simulated benchmarks, the multimodal C-BeT variant outperforms BeT and the other listed baselines across CARLA, BlockPush, and Kitchen, with an average 45.7% improvement over prior state-of-the-art work. The abbreviated table in the source reports BeT at $k$2 on CARLA, $k$3 on BlockPush, and $k$4 on Kitchen, while multimodal C-BeT reaches $k$5, $k$6, and $k$7, respectively. The same paper further reports that useful task-centric behaviors can be learned on a real-world robot purely from play data, using 4.5 hours of uncurated Franka kitchen play data and no task labels or reward information (Cui et al., 2022).

A common misconception is therefore that BeT is already goal-conditioned. The literature instead separates the two: BeT provides the multi-modal continuous-action sequence model, whereas C-BeT adds future-conditioned goal specification on top of that backbone.

4. Scaling limits of $k$8-means and the transition to VQ-BeT

Later work identifies structural limitations in BeT’s $k$9-means tokenizer. Because $\{A_1,\dots,A_k\}\subset\mathcal{A}$0-means relies on Euclidean distance, it is described as unstable or ineffective in high-dimensional action spaces, poor at handling long sequences with complex correlation, sensitive to the fixed cluster count $\{A_1,\dots,A_k\}\subset\mathcal{A}$1, and unable to provide gradient flow to discrete tokens because it is non-parametric. Joint clustering of chunked or temporally extended actions further creates exponentially large spaces, which the VQ-BeT paper presents as a major scaling bottleneck for long-range actions (Lee et al., 2024).

Vector-Quantized Behavior Transformer (VQ-BeT) addresses these limitations by replacing the $\{A_1,\dots,A_k\}\subset\mathcal{A}$2-means tokenizer with a hierarchical residual vector quantization module. Action chunks $\{A_1,\dots,A_k\}\subset\mathcal{A}$3 are encoded, quantized through multiple codebook layers, and reconstructed by a decoder; primary codes capture coarse action modes and secondary codes capture finer details. Training is two-stage: first a VQ-VAE encoder-decoder is trained to tokenize actions, then a GPT-like transformer is trained to predict tokenized code sequences, optionally with a continuous offset head. The paper emphasizes that this design supports multimodal action prediction, conditional generation, and partial observations, while preserving efficient single-pass inference (Lee et al., 2024).

The shift from BeT to VQ-BeT can be summarized as follows:

Variant	Main mechanism	Representative result
BeT	$\{A_1,\dots,A_k\}\subset\mathcal{A}$4-means action bins + offsets	$\{A_1,\dots,A_k\}\subset\mathcal{A}$5 CARLA, $\{A_1,\dots,A_k\}\subset\mathcal{A}$6 Block Push, $\{A_1,\dots,A_k\}\subset\mathcal{A}$7 Kitchen (Shafiullah et al., 2022)
C-BeT	BeT with future-conditioned goal specification	Average 45.7% improvement over prior state-of-the-art work (Cui et al., 2022)
VQ-BeT	Hierarchical residual vector quantization	$\{A_1,\dots,A_k\}\subset\mathcal{A}$8 faster inference than Diffusion Policies in simulation (Lee et al., 2024)

The empirical case for VQ-BeT is broad. Across seven environments including simulated manipulation, autonomous driving, and robotics, VQ-BeT improves on state-of-the-art models such as BeT and Diffusion Policies. On unconditional tasks, it outperforms BeT, BC, and both convolutional and transformer Diffusion Policies on 5/7 simulated tasks; on Kitchen, the reported values are 3.66 for VQ-BeT, 3.07 for BeT, and 3.44 for DiffPolicy-T. On goal-conditioned Kitchen, VQ-BeT reaches 3.78 versus 3.09 for C-BeT and 3.75 for the best diffusion baseline. The same paper reports higher behavior entropy, indicating richer multimodal behavior, single-step inference times of 15.1 ms for VQ-BeT versus 98.6 ms for DiffusionPolicy-T on Kitchen, up to $\{A_1,\dots,A_k\}\subset\mathcal{A}$9 speedup on real robots, a 73% success-rate advantage over DiffusionPolicy on two-phase long-horizon robot tasks, and an average $a\in\mathbb{R}^d$0 error of 0.73 on nuScenes, compared with 0.84 for GPT-Driver and 0.74 for Agent-Driver (Lee et al., 2024).

A plausible implication is that VQ-BeT preserves the original BeT thesis—that behavior generation benefits from tokenized multimodality—but replaces a brittle fixed clustering stage with a learned, hierarchical latent action model that scales better to long-horizon and high-dimensional regimes.

5. Adaptation, residual correction, and token-based control interfaces

One line of work uses BeT as a pretrained sequential prior and augments it with online adaptation. BeTAIL combines a BeT policy learned from human demonstrations with online Adversarial Imitation Learning. In this framework, BeT provides a base action prediction from a history window of $a\in\mathbb{R}^d$1 states and actions, trained offline by minimizing MSE on demonstration sub-trajectories, while a residual policy is learned online to correct for out-of-distribution states or shifts in environment dynamics. The combined action is

$a\in\mathbb{R}^d$2

where $a\in\mathbb{R}^d$3 is the BeT action and $a\in\mathbb{R}^d$4 is the residual correction. The paper reports that this residual BeTAIL reduces environment interactions and improves racing performance and stability, including when the BeT is pretrained on different tracks than downstream learning (Weaver et al., 2024).

BeTAIL is notable because its use of BeT is explicitly unimodal and deterministic in that setting. This underscores that “Behavior Transformer” in the literature can refer either to the original multimodal discretization framework or to a transformer policy architecture deployed in a different imitation-learning stack. The racing paper’s central claim is not that BeT alone solves distribution shift, but that a frozen sequential prior plus online AIL residual can reconcile human-like sequential structure with adaptation to changed dynamics (Weaver et al., 2024).

A second line of work generalizes the token interface rather than the policy head. “Task Tokens” proposes learning a lightweight task-specific encoder that outputs extra tokens appended to the input sequence of a frozen transformer-based behavior foundation model. The transformer is kept frozen, only the task encoder is trained with PPO, and the encoder produces a task token $a\in\mathbb{R}^d$5. The reported scale is about 200k parameters per task, contrasted with 9M–25M parameters for fine-tuning the full model. The paper reports convergence $a\in\mathbb{R}^d$6 faster than alternatives such as PULSE and 46.5$a\in\mathbb{R}^d$7 fewer parameters, while preserving compatibility with prior tokens such as text or joint-condition tokens and maintaining strong OOD robustness (Vainshtein et al., 28 Mar 2025).

This suggests a broader architectural legacy of BeT: once behavior is represented as a transformer sequence with semantically meaningful tokens, adaptation can be implemented by modifying the token stream rather than the full policy. That claim is interpretive, but it is consistent with the later emphasis on task tokens, prior tokens, and frozen transformer backbones (Vainshtein et al., 28 Mar 2025).

6. Robustness, adversarial vulnerability, and open problems

Adversarial robustness has become a major concern for modern behavior-cloning policies, and the most explicit evidence in the provided literature concerns VQ-BET. A 2025 study evaluates BC, LSTM-GMM, IBC, Diffusion Policy, and VQ-BET under untargeted, targeted, and universal adversarial perturbations in a white-box deployment-time threat model. Because VQ-BET is an explicit policy, it can be attacked in the same manner as standard vision models using PGD and UAP-style procedures. The untargeted and targeted attacks are framed as maximizing or minimizing the VQ loss under an $a\in\mathbb{R}^d$8 budget, and universal perturbations satisfy $a\in\mathbb{R}^d$9 (Patil et al., 6 Feb 2025).

The study reports that VQ-BET exhibited the highest vulnerability among all tested methods across Lift, Can, Square, and Push-T, and across untargeted, targeted, and universal attacks. Under strong PGD and UAP attacks, its task success rate dropped to 0–10% on many tasks; even at $$\lfloor a \rfloor := \arg\min_i \|a - A_i\|_2, \qquad \langle a \rangle := a - A_{\lfloor a \rfloor}, \qquad a = A_{\lfloor a \rfloor} + \langle a \rangle.$$0, about 1.5% of input range, performance dropped substantially. For the Lift task under PGD with $$\lfloor a \rfloor := \arg\min_i \|a - A_i\|_2, \qquad \langle a \rangle := a - A_{\lfloor a \rfloor}, \qquad a = A_{\lfloor a \rfloor} + \langle a \rangle.$$1, the reported success rates are approximately 0.08 for attacked VQ-BET, approximately 1.0 for the original VQ-BET, and approximately 0.25 for attacked Diffusion Policy. On Push-T under PGD, the reported attacked values are 0.10 for VQ-BET and 0.23 for Diffusion Policy. Randomized smoothing improves robustness only partially; on Lift, PGD plus smoothing yields 0.66 for VQ-BET, still significantly below nominal performance (Patil et al., 6 Feb 2025).

The paper hypothesizes that the discrete nature of VQ-BET’s hierarchical action codebook leads to discontinuous decision boundaries, making the model more brittle to small perturbations than policies with smooth, continuous output spaces. That is a hypothesis rather than an established theorem, but it aligns with the reported transferability of attacks across algorithms, architectures, and tasks, and with the finding that universal perturbations can be as effective as input-specific PGD attacks (Patil et al., 6 Feb 2025).

The open research directions identified there are specific: defenses tailored to explicit, quantized action BC models; better robustness metrics beyond task success rates; smoothed or more robust quantization for action codes; hybrid policies blending continuous-space smoothness with VQ expressiveness; task-dependent robustness evaluation; and robustness training that accounts for cross-policy and cross-architecture transfer (Patil et al., 6 Feb 2025). In that sense, BeT and its descendants now occupy a dual position in the literature: they are strong models for multimodal behavior generation, but they also expose how tokenized action interfaces can create distinctive robustness and deployment challenges.