AccelSync: Verifying Synchronization Coverage in Accelerator Pipeline Programs

Abstract: AI accelerator operators are compiled into multi-stage pipeline programs where DMA, vector, matrix, and scalar units execute concurrently on shared on-chip buffers. A missing or misplaced synchronization primitive introduces hardware-visible data races that escape both simulation and golden testing, because neither models the accelerator's cross-unit visibility semantics. We formalize accelerator pipeline programs as a restricted concurrent language, define a parameterized hardware event semantics with three ordering relations -- program order, synchronization order, and barrier order -- and reduce the correctness question to barrier sufficiency: whether every cross-unit write-read pair on the same buffer is ordered by happens-before. Here "barrier" denotes an abstract ordering primitive in the model, covering vendor pipe barriers, hard-event synchronization, and equivalent frontend-normalized synchronization points. We prove that barrier sufficiency is decidable in $O(|E|^2)$ time and that our checker is both sound and complete under the modeled semantics. We implement AccelSync, a static verification tool instantiated for Ascend 910B2 and Cambricon MLU370 by changing only the hardware model. On 6,292 production kernels from the CANN operator library, AccelSync identifies 3 previously unknown synchronization hazards -- one matching a hazard class for which we observed nondeterministic outputs on Ascend 910B2 under a specific toolkit/driver configuration (CANN 8.0.RC3), though this observation was not reproducible after a subsequent driver upgrade -- and on 120 LLM-generated kernels it flags a 19.2% defect rate (95% CI: [13.0%, 27.4%]). A mutation study on 688 non-equivalent mutants yields 100% detection, and a head-to-head comparison shows AccelSync detects hazards that Huawei's runtime sanitizer msSanitizer misses, at 400x lower cost per kernel.

• The paper introduces a formal event-based model that verifies hardware-level synchronization in accelerator pipeline programs with soundness and completeness.
• It employs a three-pass hardware-behavior graph construction and parametrizes synchronization primitives to efficiently detect cross-unit hazards.
• Experimental results on production and LLM-generated kernels highlight defect rates up to 19.2%, demonstrating the tool’s practical impact in CI pipelines.

Static Verification of Synchronization Coverage in Accelerator Pipeline Programs

Motivation and Problem Scope

Current AI accelerator architectures employ multi-stage pipelines (e.g., DMA, vector, matrix, and scalar units) that execute concurrently on shared on-chip buffers, with operator compilers translating high-level tensor programs into such pipelines. While functional correctness can be preserved during compilation, the hardware-level synchronization semantics required for correct cross-unit buffer visibility can be silently violated. This risk is pronounced as existing simulation environments and golden testing on x86 do not account for the hardware's cross-unit visibility, allowing latent data races to escape detection.

Prior efforts in GPU and CPU memory-model verification, as well as recent LLM-based program verification, do not precisely model or check the cross-unit synchronization required by hierarchical accelerator pipelines. Dynamic race detectors and vendor runtime sanitizers, while hardware-aware, are limited by their reliance on complete software stacks and input coverage, precluding their use for static, large-scale audits. In this context, "AccelSync: Verifying Synchronization Coverage in Accelerator Pipeline Programs" (2605.07881) addresses the outstanding problem of statically verifying synchronization sufficiency (coverage) in structured accelerator kernels, introducing a framework and tool that achieves soundness and completeness relative to an explicit, parameterized hardware model.

Formalization: Hardware Event Semantics and Barrier Sufficiency

The core technical contribution is the formalization of accelerator pipeline programs as a restricted concurrent language, reflecting key structural properties of real accelerator code:

• Fixed, statically known pipeline stages
• Queue- and synchronization-primitive mediated communication
• No dynamic concurrency or thread creation
• Events extracted as program-ordered (po), synchronization-ordered (so), and barrier-ordered (bo) relations
• Absence of general-purpose shared-memory concurrency

Hardware event semantics are cast in terms of extracted events (reads, writes, EnQue, DeQue, barriers), with happens-before ($\rightarrow_{hb}$) relations constructed as the transitive closure of po, so, and bo per Lamport-style event-based memory reasoning. The synchronization correctness property is formulated as barrier sufficiency: for every cross-unit write-read pair on the same buffer, an $\rightarrow_{hb}$ path must exist. Barriers here are abstract synchronization events, allowing portability across hardware backends (e.g., pipe barriers, hard-event pairs, queue synchronization).

Crucially, AccelSync proves that under the modeled class, synchronization coverage checking is decidable in $O(|E|^2)$ time (where $|E|$ is the number of memory events), with both soundness (all reported violations are feasible) and completeness (all feasible violations are reported). This eliminates the need for full interleaving exploration, yielding polynomial complexity and allowing routine static auditing of operator-size kernels.

Implementation and Engineering

AccelSync is instantiated as a lightweight Python tool (3,200 SLOC) with three frontends:

• Ascend C source-level parser
• Ascend C IR-level analyzer
• Cambricon BANG C parser (for MLU370)

Frontends normalize kernel input into a stream of memory and synchronization events (with buffer/unit/stage/operation attribution), directly supporting structured parsing, recognition of vendor-specific primitives (including complex C++ template barriers and SetFlag/WaitFlag constructs), and conservative alias-resolution for buffer temporaries.

Given the extracted events, the HB graph is constructed in three passes for the po, so, and bo relations, with hardware specificity encapsulated in a pluggable model (parameterizing stage count, queue connections, and coverage rules for each synchronization primitive). Verification is performed via graph reachability (BFS), with practical median kernel verification times in the single-digit milliseconds.

Critical implementation challenges addressed include robust normalization of template-form synchronization primitives, event-driven hard-event synchronization, and buffer aliasing. The tool is deployable as a standalone module, auditor for public codebases, validator for LLM-generated code, or as a pre-code-emission compiler pass.

Empirical Evaluation

Public Kernel Audit

On 6,292 CANN production kernels (Ascend C), AccelSync verifies 6,289 as SAFE and reports 3 high-confidence synchronization risks (0.048%). This residual risk represents the class of hazards that escape even after extensive manual review and internal testing. Notably, most production kernels have non-trivial cross-unit accesses (over 14,000 write-read pairs verified).

Case study on cross_v2 revealed a VPU-write/Scalar-read hazard (missing V-to-S synchronization). Hardware reproduction under CANN 8.0.RC3 observed nondeterminism consistent with the predicted hazard (20/29 runs observed mismatches, deterministic when synchronization was added), but subsequent driver updates rendered the result irreproducible at the device level without clear resolution.

Mutation Testing

On 688 non-equivalent mutants derived by systematic synchronization removal or unit misattribution, AccelSync exhibits 100% detection, outperforming pattern-matching rules and runtime sanitization (msSanitizer missed all queue-mediated or non-local violations), establishing its effectiveness and semantic generality.

LLM-Generated and Cross-Hardware Results

Audit of 120 LLM-generated Ascend C kernels shows AccelSync flags a 19.2% defect rate (95% CI: [13.0%, 27.4%]), concentrated on omitted scalar/DMA synchronization, a stark contrast to the hardened production corpus.

Cross-hardware instantiation on Cambricon MLU370 processes 162 BANG C kernels (PyTorch ports, vendor library, community examples) in 393 ms, with two true-unsafe cases in instructional code and isolated false-positives traceable to frontend limitations, confirming scalability and hardware-model parameterization.

Comparison with Related Approaches

• msSanitizer: Misses non-local and queue-mediated hazards, incurs 2.8–4.1s per kernel, requires full framework launch (inapplicable to standalone kernel code).
• Golden simulation: Cannot detect cross-unit hazards since it sequentializes execution.
• Model checkers (CBMC, SPIN): Suffer state space explosion or immense manual burden for even minimal kernels, verifying the superiority of AccelSync’s domain-restricted $O(|E|^2)$ approach.
• LLM-based formal frameworks (e.g., FM-Agent, Proof Wright): Only verify functional sequential correctness, oblivious to hardware concurrency and visibility.

Implications and Theoretical Impact

AccelSync rigorously separates the concern of synchronization coverage from both functional correctness and runtime anomaly detection, proposing a cross-layer contract between compiler’s pipeline topology and the hardware’s documented visibility requirements. The explicit parameterization of the hardware model enables coverage even as underlying platforms evolve (e.g., transitioning from directional to full-barrier primitives).

Practically, AccelSync renders bulk static auditing of large kernel corpora tractable, suitable for pre-commit checks, integrating into CI pipelines, or auditing LLM-generated code. The empirical finding of 0.048%–19.2% synchronization hazard incidence across real and generated code further establishes the necessity of pre-deployment static auditing.

Theoretical implications are equally significant: the proof of decidability leverages the structure of pipeline concurrency, evading the general $NP$-hardness or state space blowup of the unconstrained concurrency verification problem, and paves the way for formal pipelines integrating functional and hardware-visibility verification.

Future Directions

The formulation is currently scoped to statically structured pipeline programs as formalized; extension to classes of data-dependent synchronization or runtime-variable pipeline depths is a logical next step, subject to the maintenance of decidability.

Integration into operator compiler toolchains as a pre-emission analysis would catch hazards before code emission and deployment. Additional engineering is necessary for frontends associated with further accelerator toolchains (e.g., TPUs, IPUs), but the core checking mechanism generalizes immediately to any pipeline with parametric hardware models.

Conclusion

AccelSync establishes a sound and complete static analysis for hardware-level synchronization coverage in accelerator pipeline kernels, relying on an event-based concurrent program abstraction and a parametric, explicit hardware model. It exposes synchronization hazards not discoverable by existing testing and verification techniques, with strong empirical verification and tight complexity bounds. This work offers both an immediate practical tool aligned with modern deployment realities and a clean formal framework for ongoing research in compiler-accelerator co-design and static program verification.