Realtime-VLA FLASH: Speculative Inference Framework for Diffusion-based VLAs

Abstract: Diffusion-based vision-language-action models (dVLAs) are promising for embodied intelligence but are fundamentally limited in real-time deployment by the high latency of full inference. We propose Realtime-VLA FLASH, a speculative inference framework that eliminates most full inference calls during replanning by introducing a lightweight draft model with parallel verification via the main model's Action Expert and a phase-aware fallback mechanism that reverts to the full inference pipeline when needed. This design enables low-latency, high-frequency replanning without sacrificing reliability. Experiments show that on LIBERO, FLASH largely preserves task performance by replacing many 58.0 ms full-inference rounds with speculative rounds as fast as 7.8 ms, lowering task-level average inference latency to 19.1 ms (3.04x speedup). We additionally demonstrate effectiveness on real-world conveyor-belt sorting, highlighting its practical impact for latency-critical embodied tasks.

• The paper presents a speculative inference framework that reduces latency in diffusion-based VLAs by using a lightweight draft model to propose candidate action chunks.
• It introduces a novel flow-matching verification method that evaluates multiple denoising timesteps in parallel to ensure candidate actions match the main model’s velocity field.
• Experimental results demonstrate a 3.04× decrease in inference latency with only a marginal (≤0.3%) drop in task success, enhancing real-time robotic manipulation.

Realtime-VLA FLASH: Speculative Inference for Diffusion-based VLAs

Introduction and Motivation

Diffusion-based vision-language-action models (dVLAs) have established themselves as a robust approach for embodied intelligence, excelling at generating complex multimodal action trajectories. However, their real-time applicability in robotic manipulation is limited by substantial inference latency. Traditional dVLA deployment executes the entire, sequentially denoised inference pipeline at each replanning cycle, leading to stale action commands and compromised performance in latency-sensitive settings. Existing solutions—primarily chunked action execution—ameliorate but do not eliminate the latency bottleneck, since every replan still incurs the full cost of the multi-stage diffusion process.

The Realtime-VLA FLASH framework addresses this control-loop bottleneck by introducing a speculative inference paradigm into the continuous diffusion architecture of dVLAs. FLASH minimizes full inference calls by using a lightweight "draft" model to propose candidate action chunks, followed by parallel verification against the main model's velocity field, and a phase-aware fallback to expensive inference only when necessary. This system is designed to enable high-frequency closed-loop replanning with negligible loss in task reliability.

Figure 1: Standard synchronous dVLA inference causes stale updates due to latency (a), whereas FLASH’s speculative path enables prompt robot responses by verifying candidate actions in parallel and reverting to full inference as needed (b, c).

System Architecture

FLASH consists of two inference paths: a full path and a flash (speculative) path. The full path processes raw sensory input with an image encoder, vision-LLM (VLM) prefill, and a multi-step action denoise stage, involving computationally intensive integration of the learned ODE for continuous action generation. The flash path, in contrast, retains the image encoding but omits the VLM prefill, instead running a compact draft model built atop a single Gemma block. The draft model outputs an entire action chunk in parallel, which is then subject to a hypothesis test by the main model’s Action Expert through flow-matching-based, parallel endpoint reconstructions at key denoising timesteps.

If the drafted action chunk passes the consistency verification at all timesteps within a prescribed threshold, its largest accepted prefix is executed. If verification fails—i.e., the draft diverges from the main model—the system reverts to the full path. This mechanism dramatically reduces the frequency of expensive inference cycles.

Figure 2: Dual-path framework showing (a) the components executed along each path, and (b) the structure of the lightweight draft model for parallel action chunk prediction.

Parallel Verification via Flow Matching

The central technical innovation of FLASH is the use of flow-matching-based parallel verification to validate speculative action hypotheses in continuous control space, sidestepping the lack of explicit likelihoods or token-level probabilities as found in autoregressive decoding. Given a candidate endpoint from the draft model, FLASH linearly interpolates between Gaussian noise and this draft at specific denoising timesteps. The Action Expert evaluates these intermediate states in parallel to reconstruct the predicted trajectory endpoints. Consistency is determined by bounding the deviation between the draft and the reconstructed outputs—if all stay within threshold, the chunk is accepted; if not, execution reverts to full inference.

Figure 3: Illustration of the parallel verification process, wherein multiple key timesteps are reconstructed in parallel for comparison to the speculative draft.

Phase-aware Fallback Mechanism

Despite effective local verification, cumulative errors can accrue in critical manipulation phases, particularly near discrete transition points such as gripper switching, where fine alignment is essential. FLASH implements a phase-aware fallback: it detects these transition phases by thresholding the gripper action channel and proactively returns to the full path for high-fidelity decoding. This dual protection—local verification and transition-aware fallback—yields robust task performance across high-speed and precision-critical scenarios.

Figure 4: Trajectory comparison in a grasp-and-place task, highlighting failure due to drift in the flash path and success upon fallback near fine-placement transitions.

Figure 5: Stepwise execution visualization showing precise timing of fallback and action chunk execution.

Experimental Validation

Simulation Benchmarks

Evaluations on LIBERO benchmark suites reveal that FLASH achieves substantial inference speedups while maintaining competitive task success rates. Precisely, FLASH+Triton-$\pi_0$ reduces average per-replanning round latency from 58.0 ms to 19.1 ms (3.04× speedup) and per-action inference to 1.9 ms, with only a marginal (≤0.3%) drop in task-level success. More than two-thirds of repplanning rounds are served via the speculative path, considerably decreasing compute and memory bandwidth expenditure.

Real-World Robotic Manipulation

On a real-world UR5 robotic platform, FLASH enables successful conveyor-belt object sorting at speeds up to 15 m/min—significantly outperforming non-speculative baselines, which fail under increased latency pressure. The main failure mode for slower pipelines is stale action execution, manifesting as the robot missing fast-moving objects. In contrast, FLASH sustains robust throughput and completion rates across object geometries and speeds.

Figure 6: Conveyor-belt sorting demonstration where FLASH's low-latency control enables successful real-time grasping.

Figure 7: The UR5 robot platform utilized for live conveyor-belt sorting evaluation.

Discussion and Implications

FLASH demonstrates that speculative inference can be extended effectively to continuous-control, flow-matching dVLAs, overcoming verification challenges posed by the absence of explicit likelihoods and non-parallelizable denoising. The design is compatible with existing acceleration strategies—such as Triton kernel optimization and model compression—since it operates at the inference orchestration level rather than altering base model architectures or training procedures.

Practically, the reduction in average latency and compute overhead makes FLASH a compelling pathway for deploying dVLAs in real-time control loops on both high-end and resource-constrained hardware. The phase-aware fallback logic directly addresses the heterogeneity of trajectory phases in complex robot tasks, blending speculative speed with reliability.

A notable experimental result is the ability to achieve a 3.04× reduction in task-level inference latency while maintaining success rates within 0.3% of the original baseline. Furthermore, FLASH enables manipulation at conveyor speeds beyond the capability of synchronous non-speculative baselines, empirically verifying its impact on practical deployment.

Future Directions

Further research should investigate adaptive, trajectory-dependent verification thresholds and dynamic timestep selection for verification, replacing current heuristic settings. Integration with real-time chunking and asynchronous control architectures could further enhance closed-loop responsiveness. Resource-efficient edge deployment may benefit from FLASH’s ability to amortize memory bandwidth and power draw, extending embodied AI into new deployment regimes.

Conclusion

Realtime-VLA FLASH introduces speculative inference to diffusion-based VLAs, leveraging a lightweight draft model, flow-matching-based parallel verification, and phase-aware fallback to dramatically reduce inference latency while maintaining task performance. This framework addresses a core challenge in real-time embodied intelligence, bridging the gap between high-capacity multimodal policy models and the demands of high-frequency robotic control. Future work should explore adaptive verification strategies and integration with latency-tolerant control methods to further generalize the approach.