- The paper presents a robust, two-stage RandomForest classifier that distinguishes ML training from non-ML workloads using nine NVML counters.
- It employs extensive adversarial red-teaming to validate detection accuracy, achieving up to 98.2% accuracy even under sophisticated evasion tactics.
- The approach enables hardware-enabled governance with privacy-preserving telemetry, compelling adversaries to incur high physical and performance costs for evasion.
Detecting Hidden Machine Learning Training with Zero-Overhead Telemetry
Introduction and Motivation
The paper “Detecting Hidden ML Training With Zero-Overhead Telemetry” (2606.19262) systematically investigates the adversarial robustness of hardware-based monitoring for ML training detection on NVIDIA GPUs. In response to emerging AI governance policies requiring monitoring and reporting of compute-intensive training runs, this work addresses the technical problem of reliably distinguishing ML training workloads from inference and non-ML workloads under adversarial evasion attempts. The approach focuses on the use of content-agnostic, privacy-preserving, always-available NVIDIA Management Library (NVML) telemetry signals, circumventing prior proposals with prohibitive runtime overhead or non-robustness to adversarial strategies.
Methodology
The authors employ a telemetry-based workload classification protocol using nine NVML counters (GPU and memory utilization, memory usage, power draw, temperature, SM/memory clocks, PCIe bandwidths) polled at 1 Hz, across nine GPU models spanning four architecture generations (Ampere through Blackwell). The core classifier is a two-stage RandomForest model (400 trees) distinguishing ML from non-ML, then training from inference, exploiting both static “roofline” and temporal features, including physics-based signatures (e.g., cumulative energy, power–memory correlation).
A rigorous adversarial training and red-teaming protocol is adopted. Across five rounds of monitor–evader iteration, the study introduces and defeats 20+ families of evasion tactics, encompassing utilization modulation, memory minimization, non-ML workload mimicry, online learning and dilution, multi-GPU structuring, and white-box attacks (evaders with full classifier access). The adversarial workload library is extensive: 162 unique workloads (867 training, 352 inference, 185 non-ML runs), totaling over 445,000 telemetry readings and 100+ GPU-hours. Cross-architecture robustness is evaluated extensively, including “leave-one-GPU-out” experiments.
Results
The RandomForest classifier, utilizing purely NVML features, reaches 98.2% accuracy in the binary training-vs-non-training task across the entire corpus, with macro-F1 of 0.982. In the adversarial leave-one-strategy-out (LOO) setting, mean accuracy is 95.2% on unseen evasion tactics, and 99.6–100% on all multi-GPU adversarial categories. Even against the most challenging white-box attacks, unseen adversarial strategies achieve only 43–87% evasion against the hardened classifier.
Single-GPU evasion strategies are largely defeated after targeted hardening; only certain white-box LoRA fine-tuning strategies can degrade detection below 50%. However, these hold-out attacks incur an unavoidable performance penalty or severe workload dilution, which is numerically quantified and empirically characterized as fundamentally reliant on physical constraints (backward pass energy, communication patterns, etc.). Multi-GPU evasion is not viable at any robust detection threshold, due to the elevated distinguishability of distributed training via interconnect traffic and physical utilization patterns.
The study further validates the content-agnostic, privacy-preserving nature of NVML telemetry. No model weights, training data, or workload code are required or revealed. Detection latency is near-optimal: 98.15% accuracy is achieved with a 5s telemetry window, and 98.2% with 30s windows. The classifier generalizes across held-out GPU models (including datacenter and consumer classes), with 100% detection in the LOGO regime for eight out of nine GPUs.
Technical and Governance Implications
This work establishes that robust, zero-overhead ML training classification is technically feasible using only privileged access to trusted, hardware-level telemetry streams. The practical implication is that hardware-enabled governance mechanisms (HEMs) supporting tamper-resistant telemetry readback and authenticated reporting channels can form a foundational enforcement primitive for compute governance schemes and thresholding requirements like those in recent US and international regulatory proposals.
The results demonstrate that, once the defender has access to reliable telemetry and can iterate in the classifier–evader arms race, the economic and physical cost for meaningful evasion becomes prohibitive. The only remaining attack surface is hardware- or firmware-level tampering; thus, the governance problem is relocated from model and runtime detection to hardware attestation and chain-of-trust enforcement. For privacy, the approach avoids inspection of model content, instead operating on the unavoidable physical signatures of supervised training workloads.
The cross-architecture generalization indicates that a small, representative dataset suffices for broad deployment, eliminating the need for per-site calibration. The legal, regulatory, and operational burden on providers is minimized, with strong privacy guarantees.
Limitations and Future Directions
The present evaluation is restricted to single-node (up to 8-GPU) configurations on NVIDIA hardware and PyTorch frameworks. Prospective directions include evaluation on multi-node clusters, additional hardware vendors (AMD, Huawei, custom ML ASICs/TPUs), and adversaries with access to custom kernel-level or firmware-level manipulation. The integrity of the telemetry channel remains an essential assumption—future HEMs must guarantee attestation, anti-tampering, and protected telemetry paths.
The adversarial library is extensive but not exhaustive; continued red-teaming and open access to classifiers invites novel evasion strategies, further hardening, and potentially identifying fundamental trade-offs in detection versus workload throughput.
Conclusion
This paper advances ML governance by demonstrating that physics-grounded, zero-overhead telemetry can support highly robust, privacy-preserving detection of covert training, relegating evasion to high-cost, physically-constrained regimes. The operationalization of compute governance is thus reduced to ensuring hardware-level attestation and telemetry channel integrity. The released classifier and workload corpus provide practical benchmarks and a red-teaming foundation for ongoing research.
The study’s comprehensive evidence base and empirical rigor shift future focus toward hardware-enabled mechanisms and attestation protocols, with firm implications for both national and international AI governance, chip licensing, and compliance frameworks.
