- The paper presents ExAI5G, a Transformer-based 5G/IoT intrusion detection framework that combines Integrated Gradients, a surrogate decision tree, and LLM-generated explanations.
- The extracted 16-rule logic set achieves 99.72% fidelity and 100% coverage, while the detector reaches 99.87% accuracy, 0.854 macro-F1, and 2.48 ms median CPU latency.
- The evaluation shows a transparency–performance trade-off: boosting models achieve higher macro-F1, while ExAI5G offers inspectable decisions, strong attribution faithfulness, and practical real-time operation despite weak rare-class results and untested adversarial robustness.
Overview and motivation
ExAI5G is a framework that couples a Transformer-based intrusion detection system (IDS) for 5G/IoT traffic with a logic-based explainable AI (XAI) pipeline, targeting the well-known tension between detection performance and operational transparency. The authors argue that in security-critical domains, marginal metric gains from opaque models do not justify the loss of verifiability, and they position the framework as evidence that near-state-of-the-art accuracy can coexist with fully inspectable decision logic (2604.18052).
The work departs from post-hoc feature-importance methods such as LIME and SHAP, whose instability on similar datasets has been documented as a barrier to practitioner trust. Instead, ExAI5G produces global if-then rules extracted from a surrogate decision tree, complements them with Integrated Gradients attributions, and uses LLMs to render both into natural language — with a novel three-part quantitative validation of those explanations.
Framework architecture
The pipeline has four stages:
- Transformer IDS: features are embedded into 128-dimensional vectors, a learnable [CLS] token is appended, and a 6-layer, 8-head Transformer encoder feeds a linear classifier over nine classes. Training uses focal loss with class weights to handle severe imbalance, AdamW optimization, and early stopping on validation macro-F1.
- Integrated Gradients attribution: per-instance attributions against a zero baseline are aggregated via mean absolute values into global importance rankings over 29 packet- and flow-level features spanning frame, IP, TCP, UDP, and HTTP layers.
- Surrogate decision tree and rule extraction: a CART tree (max depth 4, min leaf size 40) is trained on the Transformer's predictions as pseudolabels; each root-to-leaf path yields a logical clause. Coverage, fidelity, and redundancy (mean pairwise Jaccard among leaf support sets) quantify how faithfully the rule set approximates the deep model.
- LLM-mediated explanation and validation: four generator LLMs (Qwen2.5:14b, llama3.1:8b, phi4:14b, gemma3:27b) convert the fired rule plus top-5 attributions into bullet-point explanations, validated by semantic similarity (SentenceTransformer cosine similarity), programmatic attribution faithfulness checks (sign agreement between descriptive language and IG scores), and an actionability score from an evaluator LLM (llama3.3:70b) using a 1–5 rubric.
Preprocessing is careful about leakage: the StandardScaler is fitted only on the training split, and all experiments were repeated five times.
On a testbed-collected 5G/IoT dataset with roughly 1.75M training and 195K test records across nine classes, the Transformer achieves 99.87% accuracy and a macro-F1 of 0.854. Per-class results show perfect F1 for DDoS and DoS_MQTT, but substantially weaker performance on rare classes — Brute Force (F1 = 0.703), Device Spoofing (0.667 with only 8 test samples), and Unauthorized Data Access (0.696). This is the honest cost of class imbalance, and it means the headline 99.9% accuracy figure is dominated by benign traffic and high-volume attacks rather than uniform per-class strength.
The baseline comparison is the paper's most consequential and somewhat contradictory result: Random Forest reaches macro-F1 0.989, LightGBM 0.980, CatBoost 0.970, and XGBoost 0.966 — all well above the Transformer's 0.854. The authors explicitly concede this gap and reframe it as the central argument: the explainable system sacrifices measurable F1 for full transparency. Whether that trade-off is justified operationally is asserted rather than demonstrated with operator-performance data, though the expert study offers partial support.
The surrogate tree yields 16 rules with 99.72% ± 0.01% fidelity and 100% coverage; pruning the least-supported leaf leaves 15 rules at essentially unchanged fidelity covering 99.998% of test cases, and the eight highest-support rules alone explain over 99% of instances. These numbers are strong: a compact, human-readable rule set reproduces nearly all of the deep model's decisions, enabling deployment as firewall-style logic or forensic artifacts. Attribution analysis identifies temporal and volumetric features (frame.time_relative, tcp.time_relative, tcp.stream) as dominant, consistent with domain expectations for attack traffic.
LLM explanation quality and validation
Across 20 sampled instances, all four generators produced structurally valid output, but quality diverged:
| Generator |
Semantic similarity |
Attribution faithfulness |
Actionability (1–5) |
| Qwen2.5:14b |
0.678 |
0.99 |
4.6 |
| phi4:14b |
0.678 |
0.98 |
4.5 |
| llama3.1:8b |
0.668 |
0.94 |
4.1 |
| gemma3:27b |
0.670 |
0.91 |
3.9 |
Qwen2.5:14b and phi4:14b achieve near-perfect faithfulness, indicating reliable mapping of attribution signs to directional language ("high"/"low"). Notably, semantic similarity scores cluster tightly (~0.67) across all models despite differing downstream quality, suggesting this metric has limited discriminative power — a point the paper does not explicitly address.
A pilot study with two security analysts (20 explanations, four Likert dimensions) tempers the automatic results. Human ratings were more conservative than the evaluator LLM: actionability averaged 3.3–4.1 versus the judge's 3.9–4.6, and experts perceived larger faithfulness variation (3.3–4.1) than the automatic metric implies (0.91–0.99). Phi4:14b ranked highest by humans, while Qwen2.5:14b led automatically — model rankings differ between evaluators. The authors acknowledge the small sample precludes inter-rater agreement analysis, and the divergence indicates that LLM-as-judge scores should not be treated as substitutes for human evaluation in critical deployments.
Operational efficiency
Median inference latency is 2.48 ms per flow on CPU, supporting real-time 5G deployment without specialized hardware.
Limitations and open questions
The paper concedes three substantive limitations. First, evaluation rests on a single testbed dataset, so generalization of both performance and extracted rules to other network environments is unverified. Second, surrogate fidelity, while high, is imperfect — a small fraction of Transformer decisions remain unexplained. Third, and most pointedly, transparency itself may be exploitable: an adversary knowing the extracted rules could craft evasive traffic, and the framework's adversarial robustness has not been evaluated. Open questions include whether fidelity gaps can be closed with rule ensembles or programmatic induction without sacrificing interpretability, how explanation-aware adversarial attacks should be defended (e.g., randomized or ensemble explanations), and whether automated rule-to-response pipelines (firewall/SIEM generation) preserve correctness under adversarial pressure.
Conclusion
ExAI5G demonstrates a complete chain from deep learning detection to verified logical rules to quantitatively validated natural-language explanations, achieving 99.9% accuracy with a 16-rule set at 99.7% fidelity. Its strongest empirical claims are the high-fidelity rule extraction and near-perfect attribution faithfulness from mid-sized open-weight LLMs under constrained prompting. Its weakest points — a macro-F1 below gradient-boosting baselines, weak rare-class performance, single-dataset evaluation, untested adversarial robustness, and optimistic LLM-as-judge scoring relative to human experts — are acknowledged but unresolved. The paper's contribution is best read as a demonstration that interpretable-by-design IDS architectures are operationally viable, with the performance-versus-transparency justification remaining an argument rather than a measured outcome.
