Patatrack: GPU-Optimized CMS Tracking
- Patatrack is a GPU-optimized, rule-based track reconstruction algorithm for CMS HLT that converts compressed raw pixel data into pixel tracks and primary vertices under strict online constraints.
- It employs a multi-stage pipeline executed via 40+ GPU kernels, leveraging a cellular automaton for pattern recognition and a Triton backend for efficient concurrency and low overhead.
- Patatrack also serves as a benchmarkable mini-app for evaluating heterogeneous computing models and portability layers in high-energy physics workflows.
Searching arXiv for Patatrack and related CMS heterogeneous tracking papers. Patatrack is a GPU-optimized, rule-based track reconstruction algorithm designed for the CMS High-Level Trigger (HLT), where it reconstructs pixel tracks and primary vertices from pixel detector raw data and beam spot information under stringent online latency and throughput constraints. In the current literature, Patatrack is presented both as a CUDA-centric reconstruction pipeline and as a remotely invoked tracking service, with pattern recognition performed by a cellular automaton rather than by machine learning. Its published role is therefore simultaneously algorithmic, architectural, and operational: it is a concrete CMS tracking workload, a production-oriented heterogeneous pipeline, and a test case for service-oriented and portable execution in the High-Luminosity LHC computing regime (Zhao et al., 9 Jan 2025).
1. Definition and role within CMS online reconstruction
Within CMS, Patatrack targets online pixel reconstruction in the HLT. Its function in the 2025 service study is narrowly defined: given compressed raw pixel data and beam spot information, it produces reconstructed pixel tracks, primary vertices, and additional outputs required by downstream HLT modules such as muon reconstruction. The emphasis is on online event processing, where the relevant optimization criteria are not merely algorithmic correctness but also end-to-end latency, throughput, and GPU occupancy under multi-client demand (Zhao et al., 9 Jan 2025).
Patatrack is explicitly characterized as a rule-based algorithm. Its track pattern recognition is implemented with a cellular automaton, and the paper contrasts this with Exa.TrkX, which is treated as a machine-learning-based alternative using MLPs, GNNs, FRNN/FAISS, and WCC in the same service framework. This distinction matters because Patatrack’s computational structure is shaped by many fine-grained GPU kernels and by deterministic reconstruction stages rather than by sequential neural-network inference modules (Zhao et al., 9 Jan 2025).
A later portability-layer study further abstracts Patatrack into a representative CMS workload. There it is described as a standalone version of the CMS heterogeneous pixel reconstruction from the CMS Patatrack project, characterized by a pipeline spanning 40+ GPU kernels, multithreaded concurrency in the host-side framework and GPU submission, and use of memory pools. In that formulation, Patatrack is not only a tracking algorithm but also an application profile for evaluating GPU programming models (Atif et al., 24 Jan 2026).
2. Reconstruction pipeline and computational structure
Patatrack’s reconstruction chain, as documented in the service study, proceeds through a fixed series of stages. Compressed raw pixel data are first unpacked into digis during digitization. Neighboring digis within a layer are then clustered to form hits, or spacepoints. Seeding follows by linking hits in adjacent layers into doublets and extending these into triplets, which act as seeds. Pattern recognition and track building then assemble sets of hits that likely belong to a single trajectory. Finally, tracks are fitted to estimate track parameters, and primary vertices are reconstructed, together with the downstream products needed by the rest of the HLT workflow (Zhao et al., 9 Jan 2025).
The implementation is explicitly organized for GPU execution. The custom Triton backend wraps Patatrack’s CUDA implementation and invokes GPU kernels for each stage of the pipeline. Common detector constants, including layer positions and indices, are preloaded onto the GPU at server startup. Intermediate reconstruction objects remain resident in device memory across stages, and only those objects required by downstream HLT modules are copied back to host memory after reconstruction. This device-resident organization is central to the reported result that network transfer overhead is negligible relative to computation (Zhao et al., 9 Jan 2025).
The published discussion is systems-oriented rather than formal-algorithmic. The service paper does not provide explicit mathematical formulas for Patatrack, including helix parameterization, chi-square scoring, or Kalman filter updates. That omission should not be read as an absence of fitting or vertexing; rather, it indicates that the paper’s contribution is the execution model and service architecture, not a re-derivation of tracking mathematics (Zhao et al., 9 Jan 2025).
3. Service architecture and device-resident execution
In the service-oriented formulation, Patatrack is deployed through a client/server model in which HLT CPU threads or jobs send inference requests to a remote Triton server hosting Patatrack as a custom backend. Communication uses gRPC for request and response transport, while Triton provides the server framework, request scheduling, queuing, and model-instance management. The request payload contains the event data needed by the backend, and the backend follows Triton’s general inference pattern even though the workload is a rule-based reconstruction chain rather than a neural network (Zhao et al., 9 Jan 2025).
The execution workflow has three published phases. During server initialization, detector geometry, configuration, and other stable environment data are preloaded into GPU memory. During execution, the server receives compressed raw pixel data together with beam spot information, runs digitization, clustering, seeding, track building, and fitting/vertexing entirely on the GPU, retains intermediates on the device, and copies only the required outputs back to host memory. At termination, reconstructed tracks and vertices are returned to the client and per-request resources are cleaned up (Zhao et al., 9 Jan 2025).
The data-flow profile is asymmetric. Inputs are approximately 80 kB per event, consisting of compressed raw pixel data and beam spot information, whereas outputs are approximately 2 MB per event, including tracks, vertices, and necessary intermediate products for downstream HLT modules. Triton handles serialization and deserialization across the network, while the backend avoids unnecessary type conversions and host-device copies within the server. In standalone Patatrack tests, the server sustains output data rates of approximately 1 GB/s with no notable penalty relative to direct GPU execution (Zhao et al., 9 Jan 2025).
Concurrency is a first-class design feature. Multiple CPU cores issue requests concurrently, and Triton schedules them through instance-level concurrency; the paper explicitly tests 10 model instances on a single T4 for saturation studies. In production HLT, the pipeline is asynchronous: tracking runs remotely while other HLT modules execute on the client CPU. This avoids idle waiting and is presented as an implemented production setup rather than as a purely conceptual diagram (Zhao et al., 9 Jan 2025).
4. Throughput, latency, and saturation behavior
The service study reports that Patatrack-as-a-service retains the performance of direct GPU offload while substantially increasing the number of concurrent CPU clients that a single GPU can serve. For a standalone deployment on an NVIDIA T4, a single-threaded client reaches 400 events/s, while a 10-thread client reaches 820 events/s; the CPU-only reference is approximately 25 events/s on a single 4-threaded HLT job. The paper observes no significant throughput difference between direct GPU and as-a-service execution in the single-thread case, and attributes the more than twofold increase from 400 to 820 events/s to improved concurrency that keeps the GPU busier (Zhao et al., 9 Jan 2025).
| Scenario | Configuration | Reported result |
|---|---|---|
| Standalone T4 | Single-threaded client | 400 events/s |
| Standalone T4 | 10-thread client | 820 events/s |
| CPU-only reference | Single 4-threaded HLT job | ~25 events/s |
| Integrated HLT, direct GPU offload | Up to 64 CPU cores | ~10% throughput gain vs CPU-only |
| Integrated HLT, Patatrack-as-a-service | In excess of 120 CPU cores | Same ~10% throughput gain |
| Saturation study, single T4 with 10 instances | Up to ~240 synchronized 4-thread client jobs | Maximum throughput maintained; declines beyond that |
In the integrated HLT workflow, the comparison is not expressed primarily in events per second but in system throughput relative to the CPU-only baseline. The direct GPU offload model used by CMS HLT is reported to be limited to 64 CPU cores requesting Patatrack inference while delivering an approximately 10% throughput gain over CPU-only execution. Patatrack-as-a-service maintains that same approximately 10% gain while servicing in excess of 120 CPU cores, and thus more than doubles the number of threads a single GPU can support relative to the pre-existing direct-attached model. Repeated measurements were stable, uncertainties were taken as the standard deviation across runs, and an approximately 2% throughput increase was observed for as-a-service relative to direct GPU connection at comparable load (Zhao et al., 9 Jan 2025).
Latency behavior is reported as equally important. The paper states that Patatrack-as-a-service processes requests from multiple CPU cores concurrently without increasing per-request latency. Because the HLT pipeline is asynchronous, remote tracking overlaps with other client-side work, so the service abstraction is presented as preserving end-to-end latency while improving GPU utilization (Zhao et al., 9 Jan 2025).
5. Patatrack as a benchmarkable HEP mini-app
In the HEP-CCE packaging and benchmarking effort, Patatrack is one of several heterogeneous HEP mini-apps selected alongside p2r, FastCaloSim, and the WireCell Toolkit. The suite is intended to exercise a broad range of GPU characteristics and to support cross-platform and facility benchmarking and evaluation. Within that suite, Patatrack represents GPU-centric tracking and reconstruction workloads that stress execution patterns characteristic of real HEP event reconstruction (Atif et al., 13 May 2025).
The 2025 HEP-CCE paper does not document Patatrack’s internal kernels, tracking stages, data structures, portability implementation, container recipes, Spack specifications, datasets, or quantitative performance results. Instead, it develops turn-key deployment methodology on other mini-apps and treats Patatrack as a target to which the same mechanisms would apply. The documented deployment patterns are twofold. One path uses containerization, with Docker base images for NVIDIA and AMD variants, automated build and execution, registry tagging and push, and CI-triggered benchmarking on HPC systems through a webhook, gateway validation, and a self-hosted runner. The other path uses Spack with a CMake-based build system, implementation and backend variants, and external vendor compiler/runtime packages to support native HPC module stacks (Atif et al., 13 May 2025).
This suggests that, in the HEP-CCE context, Patatrack’s principal value is as a representative workload for facility evaluation rather than as the paper’s main packaging case study. The paper’s explicit statement is that Patatrack is within scope as a target mini-app and would follow the same pipeline pattern demonstrated for FastCaloSim and p2r, including automated metrics collection and trend monitoring over time (Atif et al., 13 May 2025).
6. Implications for GPU portability-layer selection
The 2026 portability-layer study uses Patatrack to analyze how application characteristics influence the choice among Kokkos, Alpaka, SYCL, OpenMP target offload, and std::par. The study’s central observation is that Patatrack combines many short kernels, multithreaded kernel submission, memory pools, realistic HEP data-model complexity, and host-device coordination. Under those conditions, kernel-launch latency, thread-pool interoperability, explicit memory control, and atomic performance become decisive criteria rather than secondary engineering details (Atif et al., 24 Jan 2026).
Kokkos is described as offering mature abstractions and consistent memory APIs, but the study identifies several mismatches for Patatrack. Kokkos::Views can impose allocation and initialization overhead, with default initialization specifically called out as significant unless bypassed with Kokkos::ViewAllocateWithoutInitializing. More seriously for Patatrack, Kokkos can add tens of microseconds of launch latency, and many tens of microseconds on AMD GPUs, while its serial and threads backends interact poorly with external multithreaded frameworks. The study therefore treats Kokkos as comparatively less aligned with Patatrack’s many short kernels and multithreaded submission pattern (Atif et al., 24 Jan 2026).
Alpaka is presented as a stronger fit for Patatrack’s execution profile. Its explicit backend selection and fine-grained control over work division are reported to avoid the additional launch-latency penalties seen with Kokkos. The trade-off is increased build and integration complexity, because each backend must be compiled separately and multiple backend libraries loaded into one process require careful symbol management (Atif et al., 24 Jan 2026).
SYCL is described more conditionally. The study reports no large extra launch penalty relative to native backends, which is favorable for short-kernel pipelines, and notes a cross-platform path for RNG portability through oneMKL interoperability. However, queue concurrency from multiple threads is said to vary substantially across SYCL implementations and compiler versions, and AMD architectures are reported to lack robust concurrent SYCL solutions in the study’s observations. For Patatrack, SYCL is therefore attractive only if implementation-specific concurrency behavior is validated on the target hardware (Atif et al., 24 Jan 2026).
OpenMP target offload is treated as viable for kernels that fit the pragma model and as operationally attractive because Clang can compile for multiple architectures with --offload-arch and select at runtime based on discovered hardware. But the study also notes limited expressiveness for complex pipelines, incompatibilities with Eigen, and strong dependence of atomics performance on compiler version; its generic example reports atomic<int> under clang13 as approximately 35× slower than CUDA’s atomicAdd, although newer clang releases improve this substantially (Atif et al., 24 Jan 2026).
std::par is explicitly judged unsuitable for Patatrack. The study cites its reliance on Unified Shared Memory only, page-fault-driven transfers, and lack of explicit control and synchronization constructs for complex GPU pipelines. For a workload with persistent allocations, memory pools, and fine-grained submission, that limitation is fundamental rather than marginal (Atif et al., 24 Jan 2026).
7. Limits, future directions, and terminological disambiguation
The service-oriented Patatrack results are not presented as unboundedly scalable. The measured saturation point in the single-T4 study with 10 model instances occurs at approximately 240 synchronized 4-thread CPU client jobs, beyond which throughput declines. In the integrated HLT measurement, the overall throughput gain from GPU offloading Patatrack remains approximately 10%; the service model preserves that benefit while serving more clients, but it does not change the absolute magnitude of the HLT-level gain reported for that workflow (Zhao et al., 9 Jan 2025).
Several operational gaps and future directions are explicitly noted. Fault tolerance and application-level error handling are not detailed in the service paper. Proposed future work includes optimizing load balancing across multiple GPUs, extending portability to other GPU vendors and to other coprocessors such as ARM, TPUs, and FPGAs, and further reducing overhead. The broader implication is that the current literature treats Patatrack less as a finished portability endpoint than as a production-capable workload whose deployment model is still being generalized (Zhao et al., 9 Jan 2025).
A recurring source of confusion is nomenclature. Patatrack should not be conflated with PATrack, the 2026 multi-modal vision tracker based on progressive adaptation for RGB+Thermal, RGB+Depth, and RGB+Event tracking. That PATrack paper explicitly states that it is unrelated to the GPU-based high-energy physics tracking project maintained by CERN/HL-LHC communities (Wang et al., 22 Mar 2026).
Taken together, the cited works define Patatrack as a CMS pixel-tracking pipeline whose significance extends beyond its reconstruction stages. It is a concrete example of GPU-first online tracking, a demonstrator of remote inference-style execution for HLT farms, a benchmark mini-app for facility evaluation, and a demanding workload for portability-layer selection under realistic HEP concurrency and memory-management constraints.
