Agentic SABRE: An Uncertainty-Aware Neuro-Symbolic Multi-Agent Framework for Adaptive Ransomware Detection

Abstract: Ransomware has evolved into a complex, adaptive, and fast-moving adversary category in which static signatures and monolithic classifiers fail to generalise under concept drift, evasion, and behavioural polymorphism. In this paper, we present Agentic SABRE (Semantic-Behavioural Arbitration for Ransomware Evaluation), an uncertainty-aware, neuro-symbolic, multi-agent framework for adaptive ransomware detection. SABRE fuses semantic, representation-based evidence with behavioural, time-window forensic telemetry and employs Monte Carlo Dropout inference to quantify epistemic uncertainty for each agent. We introduce a decision-layer orchestrator that performs risk- and uncertainty-aware triage using two interpretable thresholds: a risk score and an uncertainty budget. High-confidence, high-risk samples are automatically contained, while uncertain or borderline cases are escalated to human analysts, establishing a flexible computational contract between autonomous response and analyst oversight. To support auditability and trust, SABRE integrates post-hoc explainability mechanisms, including gradient saliency, permutation importance, and counterfactual analysis, enabling both local and global interpretation of agent decisions. Extensive evaluation on RDset and RanSMAP demonstrates that Agentic SABRE preserves perfect discrimination on saturated semantic datasets, with AUC equal to 1.0, while improving robustness under weak behavioural signals. It achieves up to a 4.9 percent relative reduction in false escalations at equal recall while maintaining calibrated predictive uncertainty. Counterfactual analysis further shows that semantic and behavioural decisions can be reversed with bounded perturbation cost, indicating stable and interpretable decision boundaries.

• The paper presents a novel neuro-symbolic multi-agent framework that leverages semantic embeddings and behavioral telemetry with calibrated uncertainty for ransomware detection.
• The method fuses risk estimates using CTGAN augmentation and a formal triage policy to autonomously decide on containment, escalation, or allowance.
• Experimental evaluation shows enhanced discriminability, reduced false escalations, and robust performance under adversarial evasion and concept drift.

Agentic SABRE: An Uncertainty-Aware Neuro-Symbolic Multi-Agent Framework for Adaptive Ransomware Detection

Motivation and Framework Design

Ransomware presents a rapidly evolving adversarial landscape characterized by polymorphism, operational drift, and evasion techniques undermining static signature-based or monolithic ML classifiers. Agentic SABRE addresses this challenge through a neuro-symbolic, multi-agent framework combining semantic and behavioural detection modalities. Each agent is trained on structurally heterogeneous input spaces—semantic embeddings derived from PE metadata and behavioural telemetry collected from runtime system activity respectively. The architecture incorporates Monte Carlo Dropout to provide calibrated epistemic uncertainty estimates per agent. These uncertainties are incorporated, rather than merely reported, as operational constraints in a formal triage policy that governs containment, escalation, and allowance decisions.

Figure 1: Operational pipeline with uncertainty-aware, agentic policy. Semantic and behavioural CNNs (with MC Dropout) yield mean scores and per-agent uncertainties. Mean scores are concatenated and CTGAN-augmented for the Decision Agent to produce a fused risk $\hat{p}$.

The distinguishing properties include uncertainty-aware policy orchestration, explicit symbolic triage cues derived from ATT&CK tactics and heuristic signals, and score-level class imbalance mitigation via CTGAN augmentation. Unlike standard ensemble approaches, this multi-agent fusion leverages complementary modalities and their uncertainty signals for operational robustness.

Uncertainty-Aware Triage Policy

Agentic SABRE's triage policy formalizes operational decisions via two pairs of thresholds—risk $(\tau, \tau_{\text{high}})$ and uncertainty $(\kappa, \kappa_{\text{low}})$. Autonomous containment is permitted only for samples with fused risk exceeding $\tau_{\text{high}}$ and uncertainty below $\kappa_{\text{low}}$; otherwise, the system escalates to analyst oversight or allows execution for benign, confident cases. Thresholds are axis-aligned (vertical and horizontal planes in the fused score/uncertainty space), enhancing auditability and policy recalibration without retraining. The policy is conservative, with the maximum uncertainty across agents acting as a strict operational control variable.

Figure 2: Triage policy geometry in the $(\hat{p}, \sigma_{\max})$ space, partitioning ALLOW, ESCALATE, and AUTO-CONTAIN regions. Uncertainty acts as a stabilizing control variable.

This operational design tightly couples uncertainty estimates to decision-making, contrasting with prior approaches where uncertainty serves only as a diagnostic signal.

Data and Evaluation Protocol

Experiments leverage RDset (PE-based semantic data) and RanSMAP (temporal behavioural telemetry), with semantic and behavioural agents trained independently and fusion occurring only at the score level. Class imbalance is mitigated through SMOTE and CTGAN. Evaluation includes random-stratified, hardware holdout, temporal holdout, and family holdout splits, revealing the policy's behaviour under concept drift and unseen families. Calibration and ROC diagnostics validate fused risk as a probabilistic estimate suitable for downstream operational thresholds.

Figure 3: Mean AUROC with 95% confidence intervals across evaluation splits; family-holdout indicates a notable generalization gap, while temporal/hardware splits remain near baseline performance.

Numerical Performance and Robustness

Semantic agent performance saturates on RDset (accuracy/AUC = 1.0), utilized to stress-test fusion and triage mechanisms. Behavioural agent performance is limited by class overlap in telemetry statistics but effectively maintains uncertainty-modulated decision boundaries. Safety-optimal triage policy with $\tau = 0.999999$, $\kappa = 0.458$, $\tau_{\text{high}} = 0.99$, $\kappa_{\text{low}} = 0.229$ ensures containment only for extremely high-confidence samples.

Fusion consistently improves discriminability, is reliable under calibration (near-diagonal reliability diagrams), and reduces false escalations by up to 4.9% relative at equal recall. Under adversarial behavioral evasion strategies (slow-encrypt, memory-mask), the triage mechanism adapts: detection rates decrease slightly, escalation rates increase in response to elevated uncertainty, and samples are routed to analyst review.

Figure 4: Reliability diagram for fused risk $(\tau, \tau_{\text{high}})$0 with strong calibration, validating its interpretation for risk-aware triage.

Figure 5: Per-family detection rate under slow-encrypt attack ($(\tau, \tau_{\text{high}})$1); WannaCry exhibits vulnerability in line with elevated baseline uncertainty.

Explainability and Counterfactual Analysis

Permutation-importance and gradient saliency diagnostics reveal that both semantic and behavioural agents anchor decisions in interpretable ransomware-relevant features (entropy, access variance). Counterfactual perturbations show that both semantic and behavioural decisions can be flipped across boundaries with bounded deformation. In the fusion space, counterfactual trajectory is additive across modalities, establishing super-additive adversarial costs: evasion requires simultaneous manipulation of static and dynamic characteristics, aligning robustness and interpretability.

Figure 6: Permutation-based feature importance for the behavioural agent; variance and entropy statistics dominate, indicating the primary axes for ransomware discrimination.

Figure 7: Semantic counterfactual diagnostics; demonstrates minimal perturbation required for label flip and optimization steps indicating robustness of semantic boundaries.

Ablation and Computational Overheads

CTGAN score-level augmentation improves RanSMAP AUC by 2.5%; removing uncertainty inputs sharply increases false negatives. Fusion via trained MLP is empirically superior to naive averaging or single-modality reliance. Computational overhead is dominated by MC Dropout inference, which is practical for real-time SOC environments given parallelizable, lightweight CNN architectures.

Practical Implications and Future Directions

Agentic SABRE advances uncertainty-aware evidence fusion for cyber-defense by enabling safe autonomous containment and calibrated analyst escalation under concept drift and adversarial polymorphism. The modular neuro-symbolic triage design allows policy-level updates independent of agent retraining. Theoretical implications include operationalization of epistemic uncertainty as a formal decision constraint, super-additive adversarial costs, and interpretable decision surfaces across modalities.

Open challenges are modeling potential dependencies between semantic and behavioural feature spaces for joint likelihood estimation, improving MC Dropout calibration under nonstationary shift, and exploring non-axis-aligned triage boundaries via monotone learned surfaces. Adaptive thresholding and constrained adversarial attack implementations respecting binary and execution validity represent further priorities. Integration of cross-domain telemetry (network, process-level) and continual learning for agent retraining under temporal drift are promising avenues given ransomware's evolutionary trajectory.

Conclusion

Agentic SABRE demonstrates principled uncertainty-aware evidence fusion for adaptive ransomware detection. By leveraging epistemic uncertainty as a prescriptive operational variable within a neuro-symbolic multi-agent framework, the system achieves strong discriminability, robustness under adversarial evasion, and interpretable decision boundaries suitable for real-world deployment. The architectural modularity and explainability mechanisms position Agentic SABRE as a viable, extensible platform for practical autonomous cyber-defense, with clear theoretical links to calibrated AI triage under evolving threat landscapes (2607.04292).