A Latency-Aware Framework for Visuomotor Policy Learning on Industrial Robots

Abstract: Industrial robots are increasingly deployed in contact-rich construction and manufacturing tasks that involve uncertainty and long-horizon execution. While learning-based visuomotor policies offer a promising alternative to open-loop control, their deployment on industrial platforms is challenged by a large observation-execution gap caused by sensing, inference, and control latency. This gap is significantly greater than on low-latency research robots due to high-level interfaces and slower closed-loop dynamics, making execution timing a critical system-level issue. This paper presents a latency-aware framework for deploying and evaluating visuomotor policies on industrial robotic arms under realistic timing constraints. The framework integrates calibrated multimodal sensing, temporally consistent synchronization, a unified communication pipeline, and a teleoperation interface for demonstration collection. Within this framework, we introduce a latency-aware execution strategy that schedules finite-horizon, policy-predicted action sequences based on temporal feasibility, enabling asynchronous inference and execution without modifying policy architectures or training. We evaluate the framework on a contact-rich industrial assembly task while systematically varying inference latency. Using identical policies and sensing pipelines, we compare latency-aware execution with blocking and naive asynchronous baselines. Results show that latency-aware execution maintains smooth motion, compliant contact behavior, and consistent task progression across a wide range of latencies while reducing idle time and avoiding instability observed in baseline methods. These findings highlight the importance of explicitly handling latency for reliable closed-loop deployment of visuomotor policies on industrial robots.

• The paper introduces a latency-aware execution scheduler that calibrates observation and execution delays, then time-aligns buffered action sequences without changing the policy architecture or robot controller.
• Latency-aware execution remained stable across 100–500 ms inference delays, keeping task duration near 12 seconds, motion jerk near the 0.47 m/s³ reference, and idle time below 6%.
• The method avoided the baseline failures of blocking execution, which reached 43.8% idle time, and naive asynchronous execution, which produced contact forces up to 329 N and force smoothness above 1,150 N/s.

Motivation and problem statement

This paper addresses the observation–execution gap as a dominant systems-level constraint when deploying learning-based visuomotor policies on industrial robotic arms. The gap arises from three latency components: observation latency (sensor exposure, processing, middleware buffering), inference latency (policy computation), and execution latency (command buffering, controller dynamics, actuator response). On tabletop research platforms with low-level, high-frequency control loops, these delays are small relative to the control cycle and are often implicitly absorbed by action chunking or asynchronous execution. Industrial arms, by contrast, expose only safety-certified high-level interfaces with limited update rates and exhibit slow closed-loop response due to high inertia, long-reach kinematics, and conservative gains. The authors argue that under these conditions, timing misalignment—not policy capacity—is frequently the binding constraint on closed-loop performance.

The work is positioned against two strands of prior art. First, latency-aware training approaches that condition policies on a fixed observation–action delay [liao2025] and asynchronous chunking methods such as inpainting-based regeneration [black2025] improve temporal alignment but assume low-latency stacks or add execution-time computation. Second, prior demonstrations of robot learning on industrial arms [liu2022, chu2025] treat execution timing as a fixed system property. The paper's contribution is a reproducible system-level framework plus an execution strategy that operates purely at the interface between the policy and the robot controller, requiring no modification of policy architecture or training.

Framework and platform

The experimental platform consists of two six-axis ABB IRB 4600 manipulators (40 kg payload, 2.55 m reach) on linear tracks, controlled by an IRC5 controller using the Externally Guided Motion (EGM) Cartesian setpoint interface over UDP. Peripheral I/O is aggregated through PLCs communicating over a fieldbus, and ROS 2 provides publish–subscribe middleware between the PLC network and the workstation. This architecture deliberately isolates safety-critical real-time execution from sensing, logging, and inference processes.

Latency calibration is performed offline per modality. Execution latency, estimated via trajectory alignment of commanded versus tracked constant-velocity Cartesian motion, was approximately 225 ms. Camera observation latency, measured with a QR-code timestamp method adapted from [chi2024b], was approximately 82 ms. These calibrated values directly parameterize the execution scheduler, which is a notable strength: the framework's correctness depends on accurate latency estimates rather than generic assumptions.

Sensing comprises proprioception (TCP pose at ~83 Hz, converted from quaternions to continuous 6D rotation representation [zhou2019]), a wrist-mounted six-axis force/torque sensor resampled at 60 Hz with explicit rigid-body gravity compensation of the end-effector and grasped object, and an eye-in-hand RGB camera encoded with a fixed HOG feature extractor producing a 600D vector. The choice of a non-learned visual encoder is justified as a means of making inference latency deterministic so that scheduling effects can be isolated from feature-extractor variability. Expert demonstrations are collected through VR teleoperation implemented as relative SE(3) motion mapping with independent rotational and translational scaling, which avoids requiring extrinsic calibration between the VR tracking frame and the robot world frame.

Latency-aware execution strategy

The core mechanism is time-aligned scheduling of finite-horizon action sequences. Policy outputs $\{\mathbf{a}_{t+1}, \dots, \mathbf{a}_{t+T_p}\}$ are converted into a continuous-time buffer indexed by intended execution timestamps at spacing $\Delta\tau$. At each command update cycle, the module samples the buffered action at $\tau = \tau_{\text{now}} + \delta$, where $\delta$ is the estimated execution latency, interpolating linearly if the target falls between buffered actions. Stale actions are discarded; if no valid action exists, the previous pose is held as a safe fallback. Inference runs asynchronously and replaces the buffer upon completion. The authors are explicit that this is not a new algorithm but a systematization of latency-aware scheduling from prior systems [chi2024b], integrated into an industrial-grade pipeline—a candid framing that positions the contribution as infrastructure and evaluation methodology rather than algorithmic novelty.

Two baselines bracket the design space: blocking execution, which serializes inference and execution (avoiding stale actions but idling proportionally to inference latency), and naive asynchronous execution, which streams actions immediately without temporal compensation (minimizing idle time but executing actions outside their intended context).

Experimental evaluation

The evaluation task is contact-rich assembly of a timber corner lap joint, chosen because free-space motion tolerates latency while contact initiation and sliding do not. To isolate execution effects from learning variability, the authors deploy a deterministic k-NN policy ($T_o=1$, $T_p=16$, $k=5$) over a 615D observation space (9D pose, 6D gravity-compensated wrench, 600D HOG features), synchronized at 10 Hz. Twenty demonstrations and twenty rollouts per condition were collected; inference latency was artificially fixed at 100, 300, and 500 ms. All rollouts completed the task, so evaluation focuses on execution quality metrics—task duration, idle ratio, contact force RMS, force smoothness, and motion smoothness (Cartesian jerk)—benchmarked against the demonstration reference.

Median results across conditions:

Metric	Ref.	BE (100/300/500 ms)	NAE (100/300/500 ms)	LAE (100/300/500 ms)
Task duration (s)	13.60	18.36 / 20.72 / 23.64	9.33 / 9.03 / 9.93	12.08 / 12.18 / 11.90
Idle ratio (%)	5.10	23.40 / 37.53 / 43.78	1.17 / 4.98 / 4.48	1.42 / 4.10 / 5.80
Contact force (N)	73.98	231.71 / 56.02 / 42.72	319.71 / 180.43 / 329.32	37.99 / 35.41 / 45.28
Force smoothness (N/s)	136.06	807.67 / 54.84 / 33.15	1229.10 / 918.86 / 1153.26	41.58 / 31.12 / 33.03
Motion smoothness (m/s³)	0.47	0.85 / 0.82 / 0.70	0.61 / 0.70 / 0.66	0.47 / 0.51 / 0.46

Three findings stand out. First, blocking execution degrades monotonically with latency: idle ratio rises from 23.4% to 43.8%, roughly eight times the demonstration reference, with correspondingly degraded jerk. Second, naive asynchronous execution produces the most severe contact pathology, with contact forces up to 329 N and force smoothness exceeding 1150 N/s—roughly an order of magnitude worse than the reference—consistent with force overshoot from stale commands during contact onset. Third, latency-aware execution remains essentially invariant across the full latency range: task duration stays within about 12% of the reference at all latencies, motion smoothness matches the reference almost exactly (0.46–0.51 vs. 0.47 m/s³), and contact forces fall below the reference value in most conditions. Notably, latency-aware execution achieves near-zero idle cost (1.42% at 100 ms) comparable to naive asynchronous execution while avoiding its instability, indicating that temporal compensation recovers most of the throughput benefit of parallelism without its failure modes.

Because the policy is deterministic and all rollouts succeed, these differences can be attributed to execution scheduling rather than representational capacity or task variability—an appropriately controlled design. The authors also argue plausibly that the effect generalizes beyond k-NN inference, since any policy emitting finite-horizon action sequences faces the same staleness problem when inference latency is non-negligible.

Limitations and open questions

Several constraints bound the generality of these results. The evaluation uses a single task, a single embodiment pair, and a deliberately small, structured demonstration dataset (twenty trajectories) with fixed initial poses and translational-only control; robustness and generalization effects are explicitly excluded from scope. The k-NN proxy, while useful for isolation, does not establish behavior under stochastic or learned policies whose outputs may be less temporally coherent. Latency estimates (225 ms execution, 82 ms camera) are treated as fixed constants obtained offline; the paper does not evaluate sensitivity to latency estimation error or to time-varying latency, which is precisely the regime where methods like inpainting-based chunking [black2025] were designed to help. Validation is limited to ABB arms via EGM; other industrial platforms would require system-specific latency characterization. Finally, the interaction between latency-aware scheduling and large-scale VLAs—with substantially higher and more variable inference latency—remains untested and is identified by the authors as the key open extension.

Conclusion

This paper makes a focused systems contribution: a calibrated, reproducible pipeline for visuomotor policy deployment on industrial arms, together with evidence that time-aligned action scheduling is sufficient to maintain demonstration-consistent motion and compliant contact behavior across inference latencies from 100 to 500 ms, where both standard baselines fail in distinct ways. The central claim—that explicit handling of the observation–execution gap is necessary, not optional, for closed-loop visuomotor control on buffered industrial interfaces—is supported by controlled experiments that cleanly isolate scheduling as the causal variable. The main open question is whether these benefits persist under the higher, variable inference latencies of modern foundation-model policies and across heterogeneous industrial control architectures.