SuperSONIC Project

• The SuperSONIC Project is a cloud-native infrastructure designed to accelerate and standardize machine learning inference for large-scale, data-intensive scientific experiments.
• It leverages Kubernetes, NVIDIA Triton, and autoscaling based on real-time metrics to dynamically allocate accelerator resources like GPUs, improving efficiency over static methods.
• SuperSONIC has been successfully deployed in major experiments such as LHC (CMS, ATLAS), IceCube, and LIGO to handle complex ML workloads, demonstrating its scalability and practical utility.

The SuperSONIC Project is a cloud-native infrastructure designed to accelerate and standardize ML inference for large-scale, data-intensive scientific experiments. Built upon the SONIC (Services for Optimized Network Inference on Coprocessors) paradigm, SuperSONIC enables flexible, efficient deployment of computationally intensive inference tasks to accelerator-equipped Kubernetes clusters, with an initial focus on GPUs and deployment across globally distributed computing sites in high energy physics, astrophysics, and other domains.

1. Motivation and Requirements

Large-scale scientific experiments, such as those conducted at the LHC (Large Hadron Collider), the IceCube Neutrino Observatory, and LIGO, increasingly rely on complex ML models—including deep neural networks and graph neural networks (GNNs)—to analyze high-volume, high-velocity data streams. Traditional static allocation of coprocessors (e.g., GPUs) to local data processing nodes leads to substantial inefficiencies: resources are often underutilized during low demand, and bottlenecks emerge under fluctuating inferencing workloads. Moreover, the rapid evolution of ML models and the heterogeneity of client environments present a challenge for sustaining and updating inference infrastructure at scale.

SuperSONIC was developed to address these requirements by providing a portable, dynamically-scalable, service-oriented inference platform that can decouple client-side workflows from the server-side allocation and management of accelerator hardware.

2. System Architecture

SuperSONIC is architected as a modular, microservices-based system orchestrated via Kubernetes. The principal components and their interactions are as follows:

• NVIDIA Triton Inference Server: Hosts and executes a range of pre-trained ML models on GPU resources, exposing standard gRPC/HTTP endpoints.
• Envoy Proxy: Serves as a gateway, handling request routing, rate limiting, authentication, and load balancing between clients and inference servers.
• Autoscaler (KEDA): Monitors real-time metrics such as average request queue latency and dynamically provisions or decommissions GPU servers to match demand.

Let $N_{\mathrm{GPU}}$ denote the number of active GPU-enabled inference servers. The average queue latency is calculated as: $$L_{\text{avg}} = \frac{1}{N_{\text{GPU}}} \sum_{i=1}^{N_{\text{GPU}}} L_i$$ where $L_i$ is the request queue latency of server $i$. The autoscaler adds or removes servers based on threshold crossings of $L_{\text{avg}}$, achieving a balance between throughput and resource utilization.

On the client side, user code is modified minimally to send inference requests to a single endpoint via gRPC. The backend supports a variety of storage methods for model repositories, such as persistent disks, CVMFS, or network-attached filesystems.

3. Practical Deployment and Supported Workflows

SuperSONIC has been validated and deployed in several high-profile experimental settings:

• CMS (Compact Muon Solenoid) at LHC: Deployed for GNN and Transformer-based inference tasks.
• ATLAS (LHC): Used for ML and non-ML track reconstruction and analysis.
• IceCube Neutrino Observatory: Supports CNN workflows.
• LIGO: Used for CNN inference in gravitational wave analysis.

Deployments have ranged from single-node setups (e.g., within a GitHub Actions job) to large-scale infrastructures on clusters such as the National Research Platform (up to 100 concurrent GPU servers), and at Purdue University and the University of Chicago.

The distribution is managed via Helm charts, allowing version-controlled, declarative deployments.

4. Performance, Monitoring, and Efficiency

SuperSONIC's autoscaling capabilities have demonstrated significant practical advantages:

• Dynamic Scaling: GPU resources are automatically allocated in response to surges in inferencing demand and deallocated during lulls, markedly improving both average GPU utilization and average inference latency compared to static resource allocation.
• Versatility: The infrastructure can serve as a multi-tenant stack, supporting different experimental codes and user bases without system-level changes.
• Monitoring: Built-in instrumentation via Prometheus and Grafana provides real-time metrics for load, latency, throughput, and health status of all services.
• Load Balancing: The Envoy proxy ensures fair and efficient request distribution across available inference servers.

As an empirical benchmark, ParticleNet inference workloads (GNNs for CMS) running on NVIDIA T4 GPUs showed that dynamic resource provisioning closely tracks demand, preventing both underutilization and queuing delays.

Deployment Size	GPU Count (min–max)	Achievable Throughput	Autoscaling Response
Single-node (kind)	1	low/limited	n/a
Purdue/NRP (HPC)	1–100	proportional to load	dynamic, threshold-based

5. Challenges and Solutions

SuperSONIC addresses several key challenges in distributed inferencing:

• Client Heterogeneity: The microservice and API-layer abstraction decouples client workflows from backend details, simplifying upgrades and enabling centralized management.
• Resource Scarcity and Utilization: Autoscaling ensures that expensive accelerators (GPUs) are neither idle nor overwhelmed.
• Adoption Barriers: Use of open standards (Kubernetes, Helm, gRPC) and open-source components maximizes portability and maintainability.
• Scientific Data Movement: Integration with storage backends common in scientific computing (e.g., CVMFS).

Industry-focused alternatives such as KServe, vLLM, and native Triton deployments lacked sufficient multi-experiment support or required substantial adaptation to scientific workflows, motivating SuperSONIC’s bespoke design for research applications.

6. Future Directions

Planned expansions and anticipated evolution of SuperSONIC include:

• Broader Hardware Vendor Support: Extending beyond NVIDIA to support other GPU brands and potentially TPUs or FPGAs (with PyTriton).
• Cross-Domain Adoption: While initially focused on scientific collaborations, the architecture is suitable for broader industry and cross-disciplinary adoption wherever inference-as-a-service models are useful.
• Algorithm-Agnostic Extension: Beyond standard ML inference, the infrastructure could serve more general coprocessor-accelerated workloads, such as “Track Reconstruction as a Service.”
• Advanced Autoscaling Logic: Incorporating workload prediction and more granular resource management for improved cost/performance balance.
• Enhanced Integration: Facilitating federation across clusters and institutions for globally distributed resource sharing and maximal resilience.

7. Summary Table

Feature	Implementation in SuperSONIC	Scientific Use Cases
Infrastructure Design	Cloud-native, Kubernetes microservices, Helm charts	CMS, ATLAS, IceCube, LIGO
Model Serving	NVIDIA Triton Inference Server, gRPC endpoints	DNNs, GNNs, CNN-based tasks
Resource Allocation	Autoscaling based on real-time latency monitoring (via KEDA)	Efficient, dynamic adaptation
Load Balancing	Envoy Proxy with secure API and metrics export	Multi-experiment, multi-user
Monitoring	Prometheus and Grafana dashboards	Performance tuning, health
Adaptability	Deployable from personal workstations to supercomputers	Portability, reproducibility
Performance	Demonstrated higher average utilization and lower latency than static allocation	Cost and throughput improved
Hardware Support	NVIDIA GPUs (current), plans for extension to others via PyTriton	Prospective broadening

---

SuperSONIC enables large-scale scientific collaborations to dynamically and efficiently distribute ML inference workloads over accelerator resources, meeting the performance, scalability, and maintainability requirements of cloud-native data processing in the era of increasingly complex scientific computing. Its generalized, configurable architecture positions it for broad adoption across both research and industrial sectors as inference-as-a-service paradigms continue to proliferate.