Kunlun Anomaly Troubleshooter in LMDI

• The paper introduces KAT, a framework that integrates nanosecond-resolution kernel tracing with domain-adapted LLM causal reasoning to detect and diagnose anomalies in large model distributed inference.
• It leverages synchronized GPU agents and statistical anomaly scoring to drastically reduce diagnosis time by approximately 95% while enhancing root-cause identification by around 40%.
• KAT’s architecture unifies local Outpost agents with a global Analyzer, employing advanced instrumentation and tuned LLMs for efficient troubleshooting in production cloud environments.

Kunlun Anomaly Troubleshooter (KAT) is a comprehensive framework for anomaly detection and root cause analysis in large model distributed inference (LMDI) environments. KAT addresses the challenge of diagnosing inference performance degradation and latency variability at the kernel and system levels in GPU-centric clusters by combining nanosecond-resolution kernel tracing with domain-adapted LLM–driven causal reasoning. Evaluated in production settings at Alibaba Cloud, KAT achieves high precision and recall in anomaly detection while significantly reducing the time and manual effort required for debugging distributed inference systems (Liu et al., 8 Nov 2025).

1. System Architecture and Nanosecond-Resolution Data Collection

KAT operates in production LMDI clusters consisting of $N$ GPU-enabled servers, each hosting $M$ GPUs interconnected via NVLink/NVSwitch and Ethernet/InfiniBand for NCCL collectives. One worker process per GPU is orchestrated under standard distributed inference frameworks such as DeepSpeed or vLLM. KAT comprises two main components:

• Outpost: A local agent present on each server, encapsulating a Trace Collector (instrumenting CUDA/CUPTI, NCCL, Python runtime) and a Trace Analyzer.
• Analyzer: A global LLM inference service on a separate orchestrator, ingesting Outpost anomaly reports via gRPC.

Instrumentation leverages CUPTI for all kernel events, with LD_PRELOAD-based hooks at the Python level and NCCL collectives. Timestamps are sourced at the CPU (RDTSC, adjusted to NTP/PTP) and GPU (CUPTI-aligned, with $\lvert\delta_{clock}\rvert \lesssim 10$ ns) and normalized across workers using a “ping-pong” average offset protocol. Each trace event contains metadata such as {trace_id, PID, TID, func_name, st_ns, dur_ns, GPU_id, str_id}, and is stored in per-thread circular buffers prior to batch upload (Kafka) for local analysis.

This design enables synchronized nanosecond-resolution event streams across all distributed GPU workers, providing a rich substrate for downstream analysis.

2. Synchronicity, Consistency, and Trace Tree Alignment

KAT exploits the near-perfect synchronicity of parallel GPU worker execution: in distributed LMDI stages, all workers execute isomorphic sequences of kernels in a tightly-coupled, lock-step pattern, yielding isomorphic trace trees across devices.

Two consistency metrics are essential:

• Structural Consistency: SimHash-based Hamming distance $H(TT^{(w)}, TT^{(w')}) < \sigma_{struc}$ for trace trees $TT^{(w)}$, $TT^{(w')}$ at a given stage. Aligned trees share the same str_id.
• Temporal Consistency: Within a group of aligned events (e.g., function $f$ at layer $\ell$ across $w=1,\dots,N$ workers), durations $\{d^{(w)}_{\ell,f}\}$ should cluster tightly under normal conditions, typically forming a unimodal Gaussian-like distribution.

Normalizing clock skew ensures precise temporal comparability:

$$\hat{t}_i^{(w)} = t_i^{(w)} + \Delta C_{(w, \mathrm{ref})}$$

where $\Delta C_{(w, \mathrm{ref})}$ is the empirically measured offset for worker $w$ relative to a reference.

3. Kernel-Level Anomaly Detection

The core detection logic forms the "Trace Tree Anomaly Detection" (TTAD) routine. For each batch of structurally aligned events $G = \{d^{(w)}\}$, the anomaly score is

$S = \frac{|d - \mu_G|}{\sigma_G}$

with an anomaly flagged if $S \geq \lambda$ (a user-tunable “significance coefficient,” typically $\lambda = 3$).

Pseudocode for TTAD is as follows:

def TTAD(trace_trees, lambda_): anomalies = [] events = [root for root in trace_trees] while events: seen = set() for e in events: if e.name not in seen: seen.add(e.name) group = [w for w in events if w.name == e.name] mu, sigma = mean_stddev([w.dur for w in group]) for w in group: if abs(w.dur - mu) >= lambda_ * sigma: anomalies.append(w) events = [child for e in events for child in e.children] return anomalies

KAT leverages these anomaly marks to narrow the kernel/hardware search space rapidly.

4. Integration with Domain-Adapted LLM for Causal Reasoning

KAT fuses low-level trace-based detection with high-level reasoning:

• Prompt Engineering: Each Outpost generates JSON summaries of anomalous events, hardware metrics, and configuration, which are mapped into a canonical prompt template for the Analyzer.
• Domain-Adaptive Pre-Training (DAPT): Qwen-14B is further pre-trained for 100,000 steps on CUDA/NCCL/system logs and API documentation.
• Supervised Fine-Tuning (SFT): 26 expert-curated "hard example" prompts, with <input>, > (chain-of-thought), and <output> (succinct remediation), ensure alignment and reasoning fidelity.

  Reasoning is directed through chain-of-thought (forcing explicit causal linking, e.g., “GPU freq↓ → kernel dur↑ → NCCL wait↑ ...”) and role-based context grouping (Software/Hardware traces). The Analyzer outputs concise causal diagnoses and actionable remediation advice, filtered to maintain faithfulness and answer relevancy.

  5. Quantitative Evaluation in Production Cloud Environments

  Evaluation covers 42 production anomaly incidents (0.9–5.7M trace events/task; 200–300 threads) and targets compute, bandwidth, and storage/memory–related anomalies. Outpost is compared to rule-based and LSTM-AE baselines, while Analyzer is benchmarked against GPT-4o, Gemini-2.5-Pro, Claude Sonnet 4, and Qwen2.5-14B-Ins.

  Detection performance (Outpost):

  • Precision: 0.884

  • Recall: 0.936
  • F1: 0.901, FPR: 0.27%
  • Outperforms LSTM-AE by +26% precision, –85% false-alarm rate.

  Causal inference (Analyzer, three held-out hard cases):

  • ROUGE-L: 0.290
  • VFR: 0.970
  • $\Delta$RCA-Precision: 0.716, $\Delta$RCA-Recall: 0.360, $\Delta$RCA-F1: 0.479
  • Entailment: 0.322 (GPT-4o: 0.416, Gemini-2.5-Pro: 0.282)
  • Latency: $<2$ s for 100 K events, versus 5–10 min manual diagnosis

  KAT consistently narrows the search scope to $\lesssim$3 candidate kernels/components per anomaly, reducing operator diagnosis time by $\sim$95% and improving root-cause identification success by $\sim$40%.

  6. Technical and Operational Impact

  KAT’s combination of nanosecond-level synchronized kernel tracing with LLM-based diagnosis represents a novel methodology for achieving both high sensitivity and specificity in LMDI anomaly troubleshooting:

  • Immediate reduction in the function/hardware search space, expediting post-mortem and real-time debugging.
  • Match of precision/recall and sub-2 s diagnosis latency required for operational cloud LMDI standards.
  • Lower operator and expert labor, replacement of heuristic or highly manual root-cause processes.

  Practically, KAT demonstrates for the first time that a synergistic, trace+LLM approach can address the scale, complexity, and multi-component boundaries of distributed LLM inference environments with production-grade reliability.

  7. Implementation Guidelines and Generalization Scope

  To deploy KAT effectively:

  • Integrate Outpost on each server with hooks for CUDA, Python, and NCCL event streams. Ensure trace-buffering does not disrupt live inference.
  • Use synchronized time sources and periodic calibration for cross-device nanosecond precision.
  • Construct anomaly thresholds and SimHash parameters ($\lambda$, $\sigma_{struc}$) over a representative range of healthy inference traces.
  • When adapting the Analyzer LLM, apply DAPT and SFT with corpora that encompass the full hardware and software stack in target deployments.

  Domain adaptation is essential—use model and hardware-specific data for pre-training/fine-tuning to maximize fidelity and minimize “hallucinated” diagnoses. The same architecture can be extended (with appropriate prompt and SFT tuning) to mixed hardware and future LMDI frameworks as new performance bottlenecks emerge.

  KAT’s architectural template (fine-grained distributed tracing + LLM-guided root-cause analysis) is directly applicable to next-generation inference clusters, edge clusters with high heterogeneity, and multi-framework LMDI deployments with complex software/hardware boundary interactions.