Training-Time Action Conditioning for Efficient Real-Time Chunking (2512.05964v1)

Abstract: Real-time chunking (RTC) enables vision-language-action models (VLAs) to generate smooth, reactive robot trajectories by asynchronously predicting action chunks and conditioning on previously committed actions via inference-time inpainting. However, this inpainting method introduces computational overhead that increases inference latency. In this work, we propose a simple alternative: simulating inference delay at training time and conditioning on action prefixes directly, eliminating any inference-time overhead. Our method requires no modifications to the model architecture or robot runtime, and can be implemented with only a few additional lines of code. In simulated experiments, we find that training-time RTC outperforms inference-time RTC at higher inference delays. In real-world experiments on box building and espresso making tasks with the $π_{0.6}$ VLA, we demonstrate that training-time RTC maintains both task performance and speed parity with inference-time RTC while being computationally cheaper. Our results suggest that training-time action conditioning is a practical drop-in replacement for inference-time inpainting in real-time robot control.

• The paper presents training-time action prefix conditioning as an innovative strategy that simulates inference delays during training to remove runtime computational overhead.
• It leverages a diffusion-transformer architecture with tokenwise flow matching to maintain action continuity and enhance performance under high-latency conditions.
• Experimental results in simulation and real-world robotic tasks demonstrate improved solve rates and faster execution compared to traditional inference-time methods.

Training-Time Action Conditioning for Efficient Real-Time Chunking: An Authoritative Summary

Introduction and Motivation

The deployment of large-scale vision-language-action models (VLAs) in robotics introduces latency challenges for high-frequency, smooth, and reactive trajectory generation. Real-time chunking (RTC) mitigates this by asynchronously predicting action chunks while maintaining inter-chunk continuity via inference-time inpainting, typically increasing inference latency due to additional computational overhead. This paper proposes training-time action prefix conditioning, simulating inference delays during training rather than at inference, and producing continuity between chunks without runtime computational cost. The approach requires minimal code changes, preserves model architecture and runtime, and is applicable by fine-tuning—demonstrated with the PiZeroSix VLA on demanding real-world manipulation tasks.

Methodology: Training-Time Action Conditioning

In RTC, the controller predicts action chunks $A_t = [a_{t}, ..., a_{t+H-1}]$ with prediction horizon $H$, executing $s \leq H$ actions per chunk. Inference delay $d$ denotes the offset before an action chunk is available for execution, overlapping with previously committed actions forming the action prefix.

The conventional RTC approach conditions the model on this prefix during inference via inpainting, requiring vector-Jacobian products at each denoising step. The proposed approach pre-conditions on the action prefix during training by simulating inference delay, using ground-truth (non-noisy) prefix actions while training the model to denoise only the postfix. Architectural modifications enable tokenwise flow matching timesteps, and loss is masked solely for postfix actions.

Figure 1: Overlapping action chunks and the corresponding action prefixes (red) dictated by inference delay.

A diffusion-transformer architecture accommodates these changes with tokenwise flow matching conditioning, enabling the model to receive the exact inference delay as input, which allows for direct simulation of arbitrary delays during training.

Figure 2: Conditioning architecture for action prefix and postfix, with flow matching timesteps varying per token.

Experimental Results

Simulation Benchmarks

Experiments in the dynamic Kinetix benchmark demonstrate that training-time RTC achieves higher solve rates than inference-time RTC for considerable inference delays (i.e., $d \geq 2$), with the performance disparity increasing as the delay grows. Training-time RTC is more robust to growing prefixes, relying on explicit prefix supervision during training rather than Jacobian-based linearized inpainting during inference.

Figure 3: Simulation—solve rate vs. inference delay. Training-time RTC outperforms inference-time RTC at delays $\geq 2$.

For minimal delay ($d = 0, 1$), inference-time RTC has a marginal advantage, likely due to the reduced training supervision spent on early actions in training-time RTC.

Real-World Robotic Manipulation

The method was evaluated on two complex, sequential tasks—box building and espresso making—using the PiZeroSix VLA. Both inference-time and training-time RTC attained similar success rates and execution durations, significantly outperforming synchronous inference in terms of action speed and trajectory continuity, with training-time RTC introducing no additional runtime overhead.

Figure 4: Representative box building task for manipulation evaluation.

Figure 5: Real-world results—success rate and execution duration for box building and espresso making. Training-time and inference-time RTC perform equivalently, both faster than synchronous inference.

Training latency on H100 hardware was measured at 108ms for training-time RTC ($d \approx 5$) and 135ms for inference-time RTC ($d \approx 7$), supporting up to 200ms end-to-end latency at a 50Hz control frequency.

Implications and Limitations

The core implication is that prefix conditioning during training provides a practical means to eliminate inference-time overhead for real-time chunked action generation, crucial for high-throughput robotic control under latency constraints imposed by large VLA architectures. The drop-in nature of this method, requiring no architectural or runtime modifications, enhances its practical utility in modern benchmarks and large-scale, edge deployment scenarios.

Theoretically, the proposed method provides stronger generalization and consistency for longer action prefixes, as empirical supervision during training propagates better than gradient-based guidance at inference. However, training-time RTC is less flexible than inference-time RTC in only allowing "hard" prefix conditioning—no soft weighting or partial prefixes.

Careful simulation of expected delay distributions during training is essential, affecting generalization and adaptivity to runtime variations in latency. Future work may blend both approaches, enabling models to exploit soft prefix guidance at inference or to dynamically adapt to shifting delay profiles.

Conclusion

Training-time action prefix conditioning provides a computationally efficient, robust framework for asynchronous, real-time robot control with VLAs—addressing latency-induced discontinuities without architectural changes or inference overhead. Experimental results validate superior performance for substantial inference delays and practical parity with inference-time RTC in real-world tasks. This approach represents a practical advance for deploying high-capacity VLAs in embodied agents, with potential for future hybrid methods that unify flexibility and runtime efficiency.