Decoding Task Progress from VLA Representations

Abstract: Vision-language-action models (VLAs) are moving rapidly towards deployment as general-purpose manipulation policies, but we currently lack basic tools for understanding what these models represent internally or for monitoring them at runtime. Leveraging ideas from mechanistic interpretability, we probe the residual stream of $π_{0.5}$ and find that task progress, the normalized time remaining in a trajectory, is linearly readable from the activations. We find that this signal is present in the pretrained PaliGemma backbone prior to training on any robot-specific data. A single linear probe generalizes to unseen tasks and varies under language counterfactuals when trained on multi-prompt data, but does not enable meaningful steering of the policy. These properties make the signal directly useful for instrumenting deployed VLAs. We use the probe as a simple label-free OOD detector, which detects stalled task progress, and find it competitive with state-of-the-art methods. Our results suggest that VLAs have rich, linearly readable internal representations of semantic quantities like task progress, and that learning to read these signals offers a lightweight, interpretable path toward monitoring deployed visuomotor policies.

• The paper shows task progress is linearly readable from VLA residual streams, emerges during large-scale robotic pre-training, and generalizes across tasks with out-of-distribution R² reaching approximately 0.85 after training on eight tasks.
• The paper introduces weak decodability, strong decodability, and steerability, finding that counterfactual probe training improves instruction-sensitive decoding while progress remains not linearly steerable through single-direction activation edits.
• The paper presents a label-free progress-based OOD detector that achieves 0.960 per-episode AUROC on unseen perturbation types, outperforming supervised baselines without requiring OOD training data.

Motivation and scope

Vision-language-action (VLA) models such as $\pi_{0.5}$ are increasingly deployed as general-purpose manipulation policies, yet practitioners lack basic instrumentation for understanding what these policies represent internally or for detecting failure at runtime. This paper addresses that gap by probing the residual stream of $\pi_{0.5}$ for a single, behaviorally salient scalar: task progress, defined as the normalized fraction of the trajectory remaining, $\tau(x_t) = 1 - t/T \in [0,1]$. The authors' central claim is that progress is linearly readable from the model's activations, that the signal emerges from large-scale robotic pre-training rather than task fine-tuning, and that it supports a lightweight, supervision-free out-of-distribution (OOD) detector competitive with state-of-the-art supervised baselines (2608.13474).

The choice of progress is deliberate. Unlike the world-state features catalogued by prior probing work on VLAs—object poses, physical relations, symbolic states—progress is scalar, task-conditional, monotone along expert trajectories, and directly tied to behavior. It can be labeled for free from offline demonstration data, and its expected in-distribution behavior (monotone decrease toward zero) makes deviations immediately diagnostic.

A taxonomy: weak decodability, strong decodability, steerability

The paper's conceptual contribution is a formal distinction among three notions of "representation," moving beyond the observability/controllability dichotomy of prior work (Buurmeijer et al., 5 Mar 2026):

• Weak decodability: a linear probe achieves low in-distribution risk. This alone does not rule out spurious correlates.
• Strong decodability: the same probe, without retraining, tracks the feature under counterfactual input transformations $\mathcal{T}$ (e.g., swapping the target noun in the instruction), ruling out shortcut solutions.
• Steerability: injecting a shifted feature value along the probe direction $\bm{w}$ produces action distributions matching those elicited by the true counterfactual input, measured via a divergence $R_S$ between steered and ground-truth-counterfactual action distributions.

This taxonomy cleanly separates the existence of a linear signal from its input-dependence and its causal role in action generation—a distinction the experiments bear out sharply.

Where the progress feature lives

Probing each layer of fine-tuned $\pi_{0.5}$ (SigLIP vision encoder, 18-layer Gemma backbone, 300M action expert) with an $L_1$-trained linear probe shows progress is weakly decodable in early vision-backbone layers and the first Gemma layers. A capacity sweep against shuffled-label controls confirms the effect is selective: a large true-vs-shuffled gap persists even for high-capacity MLP probes, so the decodability is not an artifact of probe expressivity (2608.13474).

A cross-model comparison isolates the training stage responsible. Progress probes perform comparably on base $\pi_{0.5}$ (before fine-tuning), VLABench fine-tuned $\pi_{0.5}$, and fine-tuned $\pi_{0.5}$0, and all substantially outperform the PaliGemma backbone alone. The implication is that the progress feature is established by large-scale robotic pre-training, not by task fine-tuning—the pretrained VLM substrate already contains the signal prior to any robot-specific adaptation. A further control shows the feature is a property of the learned representation, not the raw input: a linear probe on raw observations collapses out of distribution (mean $\pi_{0.5}$1), while the embedding probe generalizes ($\pi_{0.5}$2).

Cross-task generalization scales with training diversity: the best task combination's out-of-distribution $\pi_{0.5}$3 rises from $\pi_{0.5}$4 with one training task to $\pi_{0.5}$5 by eight tasks, though the mean plateaus and a long negative tail indicates some task mixtures transfer poorly.

Strong decodability requires counterfactual training

The naive layer-0 probe achieves $\pi_{0.5}$6 in-distribution but is essentially invariant under noun-swap counterfactuals ("put the pear onto the plate" $\pi_{0.5}$7 "put the apple onto the plate"), indicating it reads progress from vision alone and ignores language—task progress is not strongly decodable from it. This is a notable negative result given the probe's headline in-distribution accuracy.

Counterfactual training fixes this, at a cost. Augmenting the probe dataset with shadow forward passes under swapped prompts (labeled $\pi_{0.5}$8, a proxy since true counterfactual progress is unmeasurable without separate rollouts) with a within-pair contrastive hinge and a BCE anchor yields a layer-10 probe that pins near $\pi_{0.5}$9 under swapped instructions while tracking progress normally under the original. The tradeoff is explicit: in-distribution $\tau(x_t) = 1 - t/T \in [0,1]$0 falls to $\tau(x_t) = 1 - t/T \in [0,1]$1, reflecting both the noisy proxy labels and a lower signal ceiling at layer 10. The authors are candid that the counterfactual labels are approximations and that a fully semantic definition of progress would require completion labels rather than normalized time.

Not steerable

The steerability experiments return a clean negative. Injecting $\tau(x_t) = 1 - t/T \in [0,1]$2 along the layer-0 probe direction leaves predicted progress unchanged at all downstream layers, and injected action chunks remain far from the ground-truth future actions while the non-injected policy baseline sits near zero. Task progress is therefore observable but not linearly controllable: $\tau(x_t) = 1 - t/T \in [0,1]$3's action head does not act on the progress signal in a way that single-direction, single-layer activation editing can redirect. The authors note this test is narrow—one direction, one layer, fixed magnitude—and that stronger or jointly-edited interventions remain untested. Consistent with the language-insensitivity theme, appendix experiments on the fine-tuned policy show prompt swaps are ignored in $\tau(x_t) = 1 - t/T \in [0,1]$4 episodes and position swaps are followed in only $\tau(x_t) = 1 - t/T \in [0,1]$5 episodes, indicating the fine-tuned policy acts on memorized scene layouts rather than instructions.

Progress as a label-free OOD detector

The most applied contribution is a runtime detector $\tau(x_t) = 1 - t/T \in [0,1]$6: at each replan, compare the probe's predicted progress against the expected schedule $\tau(x_t) = 1 - t/T \in [0,1]$7 and flag OOD when the residual exceeds a threshold $\tau(x_t) = 1 - t/T \in [0,1]$8. The detector requires no OOD labels, only an estimate of expected completion time.

OOD states are generated by injecting four perturbations (Gaussian noise, color shift, blur, occlusion) at three points along held-out rollouts. Under two evaluation protocols—per-replan and per-episode (max-aggregated) AUROC—the results are:

Detector	Per-replan seen	Per-replan unseen	Per-episode seen	Per-episode unseen
$\tau(x_t) = 1 - t/T \in [0,1]$9	0.796	0.833	0.910	0.851
Mahalanobis	0.986	0.859	0.775	0.719
VAE	0.859	0.796	0.782	0.778
SAFE-MLP	0.917	0.935	0.904	0.917
SAFE-LSTM	0.827	0.849	0.836	0.838

Two patterns stand out. First, $\mathcal{T}$0 is the only detector whose per-replan performance does not degrade from seen to unseen tasks (0.796 $\mathcal{T}$1 0.833), consistent with training on in-distribution data only; Mahalanobis, by contrast, is near-perfect on seen tasks but collapses on unseen ones—a memorization signature. Second, under cross-OOD-mode evaluation (two perturbation types held out from all supervised training), $\mathcal{T}$2 wins both unseen columns outright: 0.871 per-replan and 0.960 per-episode, beating both supervised SAFE detectors despite never observing any OOD example. Per-mode breakdowns show $\mathcal{T}$3 is strongest on gaussian noise (0.963) and blur (0.977) and weakest per-replan on color shift (0.750) and occlusion (0.802), where perturbations leave enough scene structure for the policy to emit plausible, progress-advancing actions; episode-level max-aggregation recovers state-of-the-art performance in all four modes. The practical implication is that progress-based monitoring is most reliable exactly at the episode-level granularity at which failure detection is operationally relevant, and it does so without the OOD supervision that SAFE requires.

Limitations and open questions

The paper's own concessions are worth stating plainly. All quantitative results come from a single model family ($\mathcal{T}$4) in simulation; transfer across VLA architectures and to real robots is untested. The progress label conflates elapsed time with semantic completion—the strong-decodability result under noun swaps provides only partial evidence against a pure-time interpretation. The counterfactual labels ($\mathcal{T}$5 for swapped prompts) are proxies, and the contrastive probe's reduced in-distribution $\mathcal{T}$6 is a real cost. The steerability negative rests on a single injection configuration. Open questions include whether richer task-conditional (non-scalar) features are similarly decodable, whether joint multi-direction edits can close the observability–controllability gap, and whether the detector's advantage holds under real-world perturbations beyond the four synthetic modes.

Conclusion

This paper establishes that task progress is a linearly readable, pre-training-derived feature of VLA residual streams that is weakly and (with counterfactual training) strongly decodable, but not steerable, and that it yields a supervision-free OOD detector competitive with supervised state-of-the-art methods—particularly under unseen perturbation types, where it attains 0.960 per-episode AUROC without any OOD data. The work contributes a useful conceptual vocabulary (weak/strong decodability, steerability) for claims about what generative policies represent, and demonstrates that lightweight probes on existing activations can instrument deployed visuomotor policies without additional models or labels.