---
title: Robot State-aware Contrastive Loss (RS-CL)
url: https://www.emergentmind.com/topics/robot-state-aware-contrastive-loss-rs-cl
type: topic
---

# Robot State-aware Contrastive Loss (RS-CL)

Robot State-aware Contrastive Loss (RS-CL) is a representation-regularization objective for vision-language-action (VLA) models that uses the robot’s proprioceptive state as a metric for shaping the embedding space of a pre-trained vision-language model (VLM), so that semantically rich visual-language features also become control-relevant for manipulation [2510.01711]. In its canonical formulation, RS-CL is trained jointly with an action-prediction objective and imposes a weighted InfoNCE loss whose pairwise weights are derived from distances in robot state space rather than from class labels, instance identity, or pure visual augmentation. The resulting representation is intended to organize observations by end-effector pose, gripper configuration, and related proprioceptive structure, thereby improving precise manipulation, especially when appearance-based VLM features remain insufficiently sensitive to robotic signals [2510.01711].

## 1. Concept and representational objective

RS-CL was introduced as a method for bridging the gap between general-purpose VLM representations and robotic control signals in VLA systems [2510.01711]. The motivating observation is that VLMs pre-trained on internet-scale vision-language data are effective at encoding semantic content, but are not trained to encode robot joint configurations, end-effector pose, gripper opening state, or the subtle spatial differences required for manipulation. In the reported visualization results, pre-trained VLM embeddings cluster primarily by visual appearance rather than by robot state, whereas manipulation requires sensitivity to “where the robot is and what it’s about to do” [2510.01711].

Within this framework, “robot state-aware” means that two time steps with similar proprioceptive states should have nearby embeddings, while states that are far apart in proprioceptive space should not be forced together. The proprioceptive signal used in the main experiments includes end-effector position, end-effector orientation in a 6D rotation representation, and gripper state; in real-world Franka experiments, absolute joint positions for the 7 joints are optionally included for finer-grained state information [2510.01711]. The representation is therefore not merely invariant to nuisance visual variation, but is explicitly shaped to reflect the geometry of robot state space.

The central objective is not to replace action prediction. Rather, RS-CL complements the standard VLA action loss with an auxiliary representation-level constraint. This distinction is essential: the action decoder continues to optimize future action prediction, while RS-CL regularizes the upstream conditioning representation so that it encodes state structure more directly [2510.01711]. A plausible implication is that the downstream decoder has less burden to recover control-relevant geometry from representations that would otherwise be dominated by scene appearance.

## 2. Formal formulation

The base VLA setting is defined at time \(t\) by multi-view RGB observations \(\mathbf{O}_t^V = \{O_t^1, O_t^2, \dots, O_t^V\}\), a language instruction \(c\), a robot proprioceptive state \(s_t\), and a future action chunk \(\mathbf{A}_t = [a_t, a_{t+1}, \dots, a_{t+H}]\) [2510.01711]. A frozen pre-trained VLM processes \((\mathbf{O}_t^V, c)\), and a lightweight adapter \(f_\phi\) maps the resulting token sequence into the conditioning representation used by the action decoder \(D_\theta\).

The action head is trained with a flow-matching objective. With \(\epsilon \sim \mathcal{N}(0, I)\), a flow time \(s \in [0,1]\), and interpolated action \(\mathbf{A}_t^s = s\,\mathbf{A}_t + (1-s)\,\epsilon\), the loss is
$$
\mathcal{L}_{\text{FM}}(\theta, \phi) =
\mathbb{E}_s \Big[
\big\| D_\theta(X, \mathbf{A}_t^s, s_t) - (\epsilon - \mathbf{A}_t) \big\|_2^2
\Big].
$$
RS-CL acts on a compact summarization of the VLM-adapter representation rather than on the full token sequence. A learnable summarization token \(u\) is appended to the VLM output, the adapter produces a corresponding embedding \(z\), and a projection head \(g_\psi\) maps it to \(h = g_\psi(z) \in \mathbb{R}^{d_{\text{proj}}}\) [2510.01711].

For a batch of size \(B\), the loss uses original embeddings \(\{h_i\}_{i=1}^B\) and augmented embeddings \(\{\tilde{h}_j\}_{j=1}^B\). The weighted InfoNCE form is
$$
\mathcal{L}_{\text{RS-CL}}
=
- \sum_{i=1}^{B} \sum_{j=1}^{B} w_{ij}
\log
\frac{\exp\!\big(\mathrm{sim}(h_i, \tilde{h}_j)/\tau\big)}
{\sum_{k=1}^{B} \exp\!\big(\mathrm{sim}(h_i, \tilde{h}_k)/\tau\big)},
$$
where \(\mathrm{sim}\) is cosine similarity and \(\tau > 0\) is the similarity temperature [2510.01711].

The distinctive component is the weight matrix. For states \(s_i\) and \(s_j\), the default metric is Euclidean distance in proprioceptive space,
$$
d_{ij} = \| s_i - s_j \|_2,
$$
and the corresponding soft supervision weights are
$$
w_{ij} =
\frac{\exp\big(- d_{ij} / \beta \big)}
{\sum_{k=1}^{B} \exp\big(- \| s_i - s_k \|_2 / \beta \big)},
$$
with \(\beta > 0\) a state-temperature [2510.01711]. Every sample pair is therefore assigned a continuous weight rather than a binary positive or negative label. Near states receive high weight; distant states receive negligible weight. This makes RS-CL a soft-label contrastive objective in which robot state similarity defines the target distribution.

The full training objective is
$$
\mathcal{L} = \mathcal{L}_{\text{FM}} + \lambda \, \mathcal{L}_{\text{RS-CL}},
$$
with \(\lambda\) initialized at \(1.0\) and cosine-decayed to \(0\) over training, so that early optimization emphasizes representation shaping and later optimization emphasizes precise action prediction [2510.01711]. The reported default hyperparameters include \(\tau = 0.2\) and \(\beta = 1.0\).

## 3. Architecture and training pipeline

The reported implementation uses GR00T N1.5 as the main VLA backbone, with a frozen VLM, a lightweight transformer adapter \(f_\phi\), and a DiT-based action decoder \(D_\theta\) trained with flow matching [2510.01711]. The same paper also reports from-scratch VLA training experiments using Qwen2.5-VL, SigLIP2, and other VLM backbones, but the architectural role of RS-CL is consistent across these settings: it regularizes the conditioning representation between the frozen or partially trainable VLM and the control head.

A notable design choice is that RS-CL does not operate on raw-image augmentations passed again through the full VLM. Instead, it uses a representation-level augmentation called **view cutoff**, applied to the VLM output. In multi-view manipulation datasets, the VLM output contains token groups associated with specific camera views. View cutoff randomly selects one view index and masks or zeroes the corresponding feature slice, producing an augmented representation \(\tilde{R}\) that still corresponds to the same underlying state and language instruction but with one camera view removed [2510.01711]. The adapter and projector are then re-run on this modified representation to obtain \(\tilde{h}\).

This design has two consequences. First, it makes the augmentation specifically aligned with multi-view robotics, where missing or degraded camera information is a realistic nuisance factor. Second, it keeps the additional compute modest, because the extra forward pass applies only to the adapter and projector, not to the heavy VLM [2510.01711]. In the reported ablations, view cutoff outperformed token cutoff, feature cutoff, span cutoff, and the absence of augmentation.

The per-batch training pipeline is thus: encode multi-view images and instruction with the VLM; append the summarization token; obtain original embeddings and decoder conditioning through the adapter; apply view cutoff to obtain augmented embeddings; compute the flow-matching action loss and the weighted contrastive loss; and optimize decoder, adapter, and projector parameters while the VLM remains frozen in the main configuration [2510.01711].

## 4. Position within robot contrastive learning

RS-CL belongs to a broader family of robot-specific contrastive objectives in which proximity in representation space is determined by task or control structure rather than by raw visual similarity. Earlier work on visual task planning compared reconstruction-based models with Siamese contrastive objectives built from same-state versus action-pair supervision and found that task-prior-driven contrastive losses outperform reconstruction-based representations under viewpoint and distractor variation [2109.06737]. In that line of work, the contrastive signal was state-aware in the sense that positives corresponded to the same underlying planning state and negatives to action-induced state changes, but the setting was latent planning rather than VLA training.

A different lineage used action-conditioned recurrent state-space models trained with an InfoNCE-like objective that contrastively predicts the next observation encoding from latent dynamics, thereby making the latent state predictive of robot dynamics and robust to camera, background, and color distractions [2112.01163]. Here, state awareness arises from action-conditioned temporal prediction rather than from explicit pair weights based on proprioceptive distance.

In manipulation imitation, CLASS—“Contrastive Learning via Action Sequence Supervision”—constructed a soft InfoNCE objective in which two observations are similar if they imply similar future action sequences, measured by Dynamic Time Warping (DTW) over action chunks [2508.01600]. CLASS can be understood exactly as a robot behavior/state-aware contrastive loss: its positive set is behavior-defined, and the strength of supervision is graded by action-sequence similarity. Relative to RS-CL in VLA models, the main difference is the supervision source. CLASS uses future action-sequence similarity, whereas RS-CL uses current proprioceptive-state distance [2508.01600].

Goal-Contrastive Rewards (GCR) introduced yet another robot-state-aware contrastive structure by combining a temporal implicit-value loss with a goal-contrastive term that pulls successful goal states together and pushes failed near-goal states away, including cross-embodiment positives [2410.19989]. This suggests a broader taxonomy: robot contrastive objectives may be conditioned by planning state equivalence, predictive dynamics, action-sequence similarity, goal-success structure, or proprioceptive geometry. RS-CL is distinguished within this taxonomy by being a lightweight end-to-end regularizer for VLA representations, using continuous soft supervision from robot state distances rather than discrete labels or temporal heuristics [2510.01711].

## 5. Empirical performance and diagnostic evidence

The main empirical evaluations covered RoboCasa-Kitchen, LIBERO, and real-world Franka manipulation tasks [2510.01711]. On RoboCasa-Kitchen with GR00T N1.5 fine-tuning, the baseline reported pick-and-place success rates of \(30.8\), \(51.8\), and \(55.3\) for 30, 100, and 300 demonstrations, with corresponding averages of \(48.2\), \(63.9\), and \(65.7\). Adding RS-CL increased pick-and-place success to \(41.5\), \(58.0\), and \(59.8\), and the overall average to \(53.0\), \(67.2\), and \(69.7\) [2510.01711]. The abstract highlights the 30-demo pick-and-place regime, where prior art was pushed from \(30.8\%\) to \(41.5\%\).

On LIBERO, the baseline GR00T N1.5 average was \(95.7\), and RS-CL increased it to \(96.4\), with the largest gain on the Long suite, from \(87.8\) to \(90.4\) [2510.01711]. These gains are smaller because the baseline is already strong, but they remain consistent.

Real-robot results were reported on four pick-and-place tasks and a close-lid task using a Franka 7-DoF arm with two cameras and 60 expert demonstrations per task. Overall success across the pick-and-place tasks increased from approximately \(45.0\%\) for the baseline GR00T N1.5 to approximately \(58.3\%\) with RS-CL, a gain of \(13.3\) percentage points [2510.01711]. The reported qualitative evidence emphasized occlusion-heavy close-lid behavior: after grasping, the wrist camera is largely occluded by the lid, yet the RS-CL model more accurately aligned the lid over the pot than the baseline.

Ablation studies are especially important for interpreting the method. Without any contrastive term, the reported RoboCasa average was \(65.7\). Standard InfoNCE without soft labels improved this to \(67.3\). Soft labels derived from next action-sequence distance and next single-action distance yielded \(66.7\) and \(66.8\), while soft labels based on current state distance achieved the best result, \(69.7\) [2510.01711]. This directly supports the claim that proprioceptive-state distance is a more effective supervisory signal than these action-based alternatives in the reported VLA setting.

The augmentation ablation showed \(65.3\) with no augmentation, \(66.3\) with token cutoff, \(67.5\) with feature cutoff, \(67.3\) with span cutoff, and \(69.7\) with view cutoff [2510.01711]. The diagnostic CKNNA analysis further showed that RS-CL significantly increased alignment between learned condition embeddings and raw proprioceptive state, corroborating that the objective does, in fact, reshape the representation toward robot state structure.

## 6. Assumptions, limitations, and interpretation

RS-CL assumes access to reliable and synchronized proprioceptive signals. The method depends on joint positions, end-effector pose, gripper state, or analogous state variables being logged accurately enough that Euclidean distance in the resulting vector space is meaningful [2510.01711]. This is not a trivial assumption: the same paper notes that it uses \(\ell_2\) distance over raw state vectors, which presumes adequate relative scaling among position, orientation, gripper, and optional joint-angle components.

The method is also evaluated primarily on manipulation benchmarks—RoboCasa-Kitchen, LIBERO, and a single-arm Franka platform—so its generality to navigation, locomotion, or multi-robot coordination is not established in the reported results [2510.01711]. Likewise, the contrastive supervision uses only proprioception; tactile sensing, force signals, object pose, and contact state are not incorporated.

A common misconception would be to regard RS-CL as a generic replacement for behavior cloning or flow matching. The reported method does not function that way. It is explicitly a regularizer added to the standard action-prediction pipeline, and its strongest reported gains appear when it complements, rather than substitutes for, the action loss [2510.01711]. Another misconception would be to view it as ordinary instance-discrimination contrastive learning. Its defining feature is precisely the use of continuous pair weights derived from robot state geometry, which avoids a purely binary positive-negative partition.

In a broader historical sense, RS-CL can be read as a concrete crystallization of a theme already present in several robot-learning literatures: contrastive objectives are most useful when similarity is defined by robot state, action, dynamics, or task outcome rather than by visual appearance alone [2109.06737; 2112.01163; 2410.19989; 2508.01600]. What distinguishes the VLA formulation is its explicit use of proprioceptive distances as soft supervision inside the standard VLA training loop, with minimal architectural intrusion and documented gains on both simulated and real manipulation [2510.01711].

Source: https://www.emergentmind.com/topics/robot-state-aware-contrastive-loss-rs-cl