Asynchronous Compute Pipeline for AI Inference

• Asynchronous compute pipeline is a design that decouples perception and generation, allowing overlapping execution and managing staleness through shared public context.
• It improves throughput by 2.54× on average in embodied AI by leveraging controlled overlap strategies and frame-based scheduling with fetch_offset adjustments.
• Its implementation employs deterministic frames, CUDA graphs, and partition tuning to optimize both auto-regressive and diffusion models while preserving accuracy.

An asynchronous compute pipeline denotes an execution organization in which logically distinct stages are decoupled and allowed to overlap under explicit control, rather than being forced through a strictly serial request-by-request loop. In embodied AI, the Auras framework formalizes this design by disaggregating perception and generation, executing them in a controlled asynchronous pipeline across frames, and using a shared public context to mitigate staleness; across six models, it reports a 2.54× average throughput improvement while achieving 102.7% of the original accuracy (Zhang et al., 11 Sep 2025).

1. Sequential embodied inference and the rationale for asynchrony

Auras starts from the standard embodied-AI workflow in which each camera frame or multimodal observation triggers a request, perception encodes that observation into a latent context, and generation then iteratively produces actions or tokens (Zhang et al., 11 Sep 2025). In the usual sequential design, only one request is “in flight” at a time. This preserves accuracy, but it leaves the system underutilized and slow.

The central systems observation is that perception and generation do not have the same computational structure. Perception is mostly one-shot encoding. Generation is iterative: auto-regressive models decode token by token, whereas diffusion policies denoise step by step (Zhang et al., 11 Sep 2025). That asymmetry creates an opportunity for overlap. The asynchronous compute pipeline in Auras therefore changes the execution model from a strict perception-to-generation loop per request into a staged pipeline in which multiple requests can occupy different stages concurrently.

A common misconception is to equate this with arbitrary concurrent execution. The paper explicitly distinguishes its design from naive multi-stream parallelism. The objective is not unconstrained concurrency, but controlled overlap that raises “thinking frequency” and GPU utilization without the chaotic interference and accuracy degradation associated with unstructured parallel execution (Zhang et al., 11 Sep 2025).

2. Disaggregation of perception and generation

Auras first analyzes the compute graph of the embodied policy and identifies the variables shared between perception and generation. It then extracts a public context that both stages can use (Zhang et al., 11 Sep 2025). This is the key abstraction that makes disaggregation operational rather than merely architectural.

For auto-regressive policies, the generation input is the embedding sequence

$X=[X_V, X_L, X_A]$

where $X_V$ is the visual embedding, $X_L$ is the language prompt embedding, and $X_A$ is the action-token embedding sequence (Zhang et al., 11 Sep 2025). Although $X_V$ and $X_L$ are fixed within a request, the paper argues that under concurrency the whole generation state becomes volatile, because different requests should not reuse a private KV-cache in the usual way. Auras therefore promotes the entire sequence $[X_V, X_L, X_A]$ to a public context. This public context can be refreshed when perception produces a new observation embedding or when generation produces new action tokens.

For diffusion policies, the shared hidden observation context $H_o$ is the volatile component made public. The iterative denoising process is written as

$$\begin{aligned} x_1&=Generate(x_0, H_o)\ &\cdots \ x_n&=Generate(x_{n-1}, H_o) \end{aligned}$$

where $x_0$ is random Gaussian noise and the denoising steps condition on the shared perception-derived context $X_V$0 (Zhang et al., 11 Sep 2025). In this case, only the perception output is used as the public context. Unlike the auto-regressive case, the paper does not apply the merged-computation trick across requests, because the denoised intermediate states are not identical across requests.

This distinction is fundamental. In the auto-regressive case, the public context includes both perception-derived and generation-evolving state. In the diffusion case, the public context is limited to the perception-derived observation context. This suggests that “asynchronous compute pipeline” is not a single fixed mechanism, but a policy-structured family of overlap strategies.

3. Public context, staleness control, and frame-based execution

The core accuracy problem in asynchronous execution is staleness: if the pipeline becomes too deep, the action produced at time $X_V$1 may actually be based on $X_V$2, skipping newer observations (Zhang et al., 11 Sep 2025). Auras addresses this with a public context buffer shared by perception and generation. Perception writes the newest context into the buffer, and generation asynchronously reads from it.

The read policy is controlled by fetch_offset. When fetch_offset = 0, perception and generation use the same frame’s public context, so they must run sequentially because of the dependency. When fetch_offset = -1, generation reads the previous frame’s context, allowing perception and generation to run in parallel asynchronously (Zhang et al., 11 Sep 2025). The paper states that fetch_offset = 0 is used for all diffusion-based policies, whereas fetch_offset = -1 is used for all auto-regressive policies.

Auras implements this design through an asynchronous pipeline executor that organizes execution into deterministic frames (Zhang et al., 11 Sep 2025). Within each frame, perception stages compute the next context and write it into the public buffer, generation stages compute on a chosen public context from a specified frame offset, and multiple frames may be active simultaneously. The pipeline degree is parameterized by $X_V$3 for perception and $X_V$4 for generation, and a request traverses a total of

$X_V$5

frames to finish (Zhang et al., 11 Sep 2025).

The paper presents the usual throughput–freshness tradeoff explicitly. Larger pipeline depth gives better overlap and throughput, but too much depth can reduce freshness and hurt accuracy (Zhang et al., 11 Sep 2025). This is why Auras prefers structured pipeline execution over unstructured parallel execution: the structured form provides more predictable scheduling, lower interference, better memory efficiency, and better opportunities to merge redundant auto-regressive computation.

4. Scheduling, partitioning, and model-specific optimization

Because the configuration space is large, Auras uses a hierarchical tuning process (Zhang et al., 11 Sep 2025). The first step grid-searches over $X_V$6 and $X_V$7, subject to an upper bound on total frame depth $X_V$8, in order to avoid obviously poor configurations. The second step tunes partitioning inside the pipeline.

The paper assumes a roughly uniform partition initially: perception stages have similar FLOP cost, and generation stages are split across stages in a balanced way (Zhang et al., 11 Sep 2025). It then introduces a skewness weight $X_V$9 for fine-tuning. The amount of computation in stage $X_L$0 is multiplied by

$X_L$1

and normalized across stages. Positive $X_L$2 pushes more work to later stages, which means generation uses fresher context more often (Zhang et al., 11 Sep 2025). Uniform partitioning maximizes throughput, whereas skewed partitioning sacrifices some throughput for better accuracy.

The diffusion results on Push-T illustrate this tradeoff concretely. A skewed partition with $X_L$3 preserves 10.25 FPS versus 11.84 FPS under uniform partitioning, but improves accuracy by 23.98% (Zhang et al., 11 Sep 2025). The parameter is therefore a direct control over the balance between frame freshness and raw pipeline saturation.

A particularly important optimization applies only to auto-regressive transformers. Because causal masking ensures that the hidden state for token $X_L$4 depends only on tokens up to $X_L$5, a larger prefill can subsume smaller ones. The paper’s claim is that overlapping auto-regressive decode steps can be fused into a single unified Generate(X) computation over the public context, substantially reducing redundant work (Zhang et al., 11 Sep 2025). This optimization is one of the reasons Auras outperforms simple multi-stream parallelism on auto-regressive policies.

5. Implementation and empirical characteristics

The Auras implementation is reported as about 4,000+ lines of Python (Zhang et al., 11 Sep 2025). Its practical mechanisms include automatic slicing of PyTorch models into multiple stages, conversion of non-PyTorch transformers into x-transformers, use of CUDA graphs to capture fixed compute patterns and reduce launch overhead, binding each CUDA graph to a specific CUDA stream to support concurrency inside frames, double-buffering of the public context buffer to avoid lock/unlock overhead, memory offloading via HuggingFace Accelerate when models do not fit entirely in GPU memory, batched execution when identical compute graphs appear within a frame, and alternating CUDA graph launches across streams to overlap CPU launch overhead.

The evaluation covers both auto-regressive and diffusion embodied agents on AMD EPYC 7763 64-core CPUs with Nvidia RTX 3090 / RTX 4090 GPUs, using CUDA 12.4 and PyTorch 2.2.1 (Zhang et al., 11 Sep 2025). Auto-regressive models include OpenVLA and RT2$X_L$6. Diffusion-based models include DP, DP-CNN, DP-plus, DP-CNN-plus, and TinyVLA. Baselines are SEQ for standard sequential perception→generation, DEC for decoupled perception and generation on separate streams, and PAR for multiple isolated workers or streams with no structured pipeline (Zhang et al., 11 Sep 2025).

The principal empirical claim is that Auras preserves accuracy while increasing inference frequency. The average success rate is 102.7% of the sequential baseline across six models (Zhang et al., 11 Sep 2025). By contrast, naive parallel execution can reduce success rate by about 80.22% on average, and in some complex diffusion cases it drops to zero. The paper attributes Auras’ stability to the public-context mechanism, which ensures that generation consumes the freshest available shared context compatible with the policy structure.

Throughput gains are reported at several granularities. Overall throughput improves by 2.54× on average, with 3.05× for auto-regressive models and 2.28× for diffusion-based models (Zhang et al., 11 Sep 2025). Against specific baselines, Auras increases execution frequency by 2.29× vs sequential, 3.01× vs decoupled, and 1.49× vs naive parallel, on average. Across all tested models and hardware, the throughput improvement ranges from 1.32×–3.48× on RTX 4090 and 1.18×–3.08× on RTX 3090. For OpenVLA specifically, the thinking frequency rises from about 6 Hz to 17 Hz (Zhang et al., 11 Sep 2025).

The structured pipeline advantage is also measured directly. For OpenVLA and RT2$X_L$7, parallel mode reaches about 1.11×, whereas Auras achieves 2.20×–3.29× (Zhang et al., 11 Sep 2025). For diffusion models, Auras exceeds the best multi-stream throughput by 1.07×–1.19×. These results support the paper’s claim that the system is not merely “more parallel,” but better scheduled.

6. Broader usage and adjacent research meanings

The phrase asynchronous compute pipeline is used in several adjacent literatures, but not always for the same object. In embodied inference, Auras uses it to denote disaggregated perception–generation execution with explicit public-context sharing (Zhang et al., 11 Sep 2025). In pipeline-parallel training, the dominant issue is stale or delayed gradients rather than stale observations.

The training literature repeatedly treats utilization and staleness as the central tension. The Nesterov-based asynchronous pipeline parallel optimization paper states that asynchronous optimization in PP is appealing because it offers 100% pipeline utilization by construction, but that weights and gradients are no longer synchronized, creating stale or delayed gradients (Ajanthan et al., 2 May 2025). PipeMare similarly defines an asynchronous PP training scheme in which the forward pass and backward pass do not necessarily use the same weights, while reporting $X_L$8 and $X_L$9 (Yang et al., 2019). AMDP addresses the same issue structurally by limiting the first stage of each pipeline to process at most two minibatches before backpropagation, yielding

$X_A$0

when $X_A$1, and it reports near-synchronous convergence together with the best throughput among the tested baselines (Chen et al., 28 May 2026). Basis-rotation work further argues that delay scales linearly with pipeline depth and reports that training a 1B-parameter LLM with basis rotation achieves the same training loss in 76.8% fewer iterations than the best-performing asynchronous pipeline parallel training baseline (Jung et al., 3 Feb 2026).

Serving and systems research use the same phrase for yet other staged decompositions. DisagFusion splits diffusion serving into Encoder, Diffusion Transformer (DiT), and Decoder services, connects them with asynchronous queues and non-blocking transfer, and reports 3.4×–20.5× throughput improvement and 18.5× lower end-to-end latency relative to a monolithic baseline (Zha et al., 25 May 2026). GPU systems work describes analogous designs at a lower level: VDCores represents workloads as dependency-connected micro-operations scheduled on resource-isolated virtual cores, Cypress compiles task-and-tensor programs into Hopper-style producer–consumer pipelines, and Tawa lowers asynchronous references into TMA plus mbarrier coordination for warp-specialized execution (He et al., 4 May 2026, Yadav et al., 9 Apr 2025, Chen et al., 16 Oct 2025).

These neighboring uses clarify a broader point. Across embodied inference, distributed training, model serving, and GPU kernel generation, an asynchronous compute pipeline is not simply “running things concurrently.” It is a staged execution model in which overlap is permitted only under explicit dependency control, and in which some mechanism—public context, look-ahead correction, bounded mismatch, delay compensation, or dependency-aware scheduling—exists to keep the overlapped system from degenerating into stale, inconsistent, or unstable computation.