Agentic-SecPBFT: Agentic AI-Driven Proactive Security Framework for Wireless PBFT Consensus in Mobile Ad-Hoc Networks

Abstract: The standard Practical Byzantine Fault Tolerance (PBFT) protocol, designed for stable, wired environments, exhibits critical vulnerabilities when deployed in settings like mobile ad-hoc networks, thus making it susceptible to sophisticated threats such as Sybil attacks, Byzantine collusion, and message manipulation. Existing static defense mechanisms are ill-equipped to handle the intelligent and coordinated nature of these attacks. To address this challenge, this paper leverages the Agentic AI paradigm to build a distributed multi-agent system in which each consensus node is equipped with an intelligent agent. These agents employ a hierarchical Multi-Agent Deep Q-Network (MADQN) algorithm to learn and execute proactive security policies in real-time. By observing local network behavior, message consistency, and dynamically maintained reputation scores, the agents collaboratively identify suspicious behavior and recommend defensive actions under standard PBFT quorum and membership rules, thereby improving the integrity of the consensus process. We refer to the resulting framework as Agentic-SecPBFT. Then, we formally model key attack vectors and conduct extensive simulations. The results demonstrate that Agentic-SecPBFT reaches a 95.0% attack detection rate with a 1.8% false positive rate. Compared with mainstream PBFT variants, it achieves 3.1* higher throughput with 56% lower latency on average under 33% malicious nodes, offering a robust and adaptive security solution for decentralized wireless systems.

• The paper’s main contribution is the design of an AI-driven hierarchical security framework that integrates local and global agents to proactively secure PBFT consensus.
• It employs a multi-agent deep Q-learning algorithm and dynamic reputation metrics to mitigate Sybil, Byzantine, and message manipulation attacks.
• Numerical results reveal a 95% attack detection rate, 56% reduced latency, and over 3× throughput improvement under challenging wireless MANET conditions.

Agentic-SecPBFT: Agentic AI-Driven Proactive Security for Wireless PBFT Consensus in Mobile Ad-Hoc Networks

Problem Context and Motivation

The deployment of Practical Byzantine Fault Tolerance (PBFT) consensus in mobile ad-hoc networks (MANETs), including VANETs and IIoT settings, is fundamentally compromised by the wireless channel's unreliability, dynamic topology, and heightened vulnerability to coordinated adversarial behavior. Sybil attacks, Byzantine collusion, and message manipulation exploit the semantic gap between logical cryptographic identities and real physical nodes, critically undermining PBFT's security assumptions ($n \geq 3f+1$). Existing static defense mechanisms are inadequate against adaptive, intelligent threats that exploit environmental noise and protocol flaws.

Figure 1: Attack models for wireless consensus networks—including (a) Sybil attack, (b) Byzantine collusion, (c) message delay and dropping—highlighting protocol vulnerabilities to coordinated adversarial strategies.

Agentic-SecPBFT Architecture

The proposed Agentic-SecPBFT framework leverages the Agentic AI paradigm to augment each consensus node with Local Consensus Agents (LCAs) and orchestrates global strategy via a Cluster Optimization Agent (COA). LCAs execute decentralized, real-time threat detection by monitoring local channel states (CSI, SINR), protocol-layer behavior, and dynamically maintained reputation scores. COA aggregates telemetry across LCAs to optimize global DQN parameters and reputation vectors, supporting cross-node anomaly detection while remaining outside PBFT's commit-critical path. The hierarchical multi-agent MADQN algorithm is used for proactive security policy learning and execution.

Figure 2: Hierarchical agent structure with LCAs for distributed local defense and COA for global optimization, facilitating centralized training and decentralized execution (CTDE).

Attack Modeling and Channel-Aware Threat Analysis

Agentic-SecPBFT formally models attack vectors—including Sybil proliferation, Byzantine collusive selective forwarding, and message delay/dropping—using probabilistic channel models (log-distance path loss, Rician fading) and mobility-induced topology evolution. The semantic gap whereby one physical node controls multiple cryptographic identities (Sybil) is addressed by integrating CSI spatial correlation and challenge-response protocols as probabilistic association evidence.

Dynamic Trust, Reputation, and Defense Policy

A dynamic reputation metric is defined, weighted by physical certainty factors and channel-aware message delivery probabilities ($P_{success}$), enabling discrimination between benign loss (fading) and malicious behavior. The MADQN algorithm features a Dec-POMDP formulation with a rich state space combining network-layer, consensus-layer, and reputation-layer features. The reward function balances security, liveness, and consensus correctness, penalizing false view changes and incentivizing maintenance of high SINR for critical message transmission. Action space for LCAs includes node flagging, reputation adjustment, challenge requests, traffic prioritization, and power control.

Figure 3: Hierarchical MADQN algorithm steps illustrating centralized COA training and distributed LCA execution, critical for scaling proactive defense in wireless MANET consensus.

Security Guarantees and Formal Analysis

Agentic-SecPBFT maintains protocol-invariant safety: block commit certificates require $2f+1$ distinct signatures within the PBFT committee, ensuring no two honest nodes commit conflicting blocks, independently of MADQN convergence and despite COA or LCA faults. Sybil mitigation relies on probabilistic identity association, with explicit bounding of residual semantic gap error ($\epsilon_{phy}$). Adaptive defensive actions cannot violate quorum rules or permanently exclude nodes unless certifiable by PBFT commit logic. Conditional liveness is assured under partial synchrony and eventual honest primary selection.

Numerical Results: Robustness, Accuracy, and Efficiency

Simulation results under diverse mobility and interference conditions demonstrate Agentic-SecPBFT's strong numerical performance:

• Attack Detection Rate (ADR): Achieves up to 95.0% ADR with only 1.8% false positive rate, significantly outperforming GRBFT, DA-PBFT, and standard PBFT even under 33% malicious nodes.
• Consensus Success Rate: Remains above 80% with high node compromise; consistently outperforms PBFT (which drops below 15%), and maintains resilience with scaling node counts and mobility.
• Latency and Throughput: Reduces consensus latency by 56%, and raises throughput by 3.1× relative to baseline PBFT under adverse attack conditions.

  Figure 4: MADQN convergence performance under four wireless scenarios; hierarchical design yields superior stability and reward maximization.

  

  Figure 5: Attack detection rate comparison under varied simulation steps, malicious node ratios, and interference intensities; Agentic-SecPBFT demonstrates robust threat identification.

  

  Figure 6: False positive rate comparison across key scenarios; Agentic-SecPBFT maintains ultra-low FPR by integrating physical layer context.

  

  Figure 7: Consensus success rate resilience across varying malicious node ratios, node count, and mobility speed; robust to network scale and dynamics.

  

  Figure 8: Consensus latency remains low under challenging network and attack conditions; Agentic-SecPBFT suppresses latency spikes from misjudged view changes.

  

  Figure 9: Consensus throughput stability; hierarchical agentic defense prevents adversarial collapse of transaction processing rates.

Ablation Study

Performance sensitivity analysis reveals physical-layer awareness (CSI/SINR context) as most critical in mitigating misjudgment and ensuring consensus stability. Dynamic reputation and COA coordination provide stabilizing effects, but the framework's safety is protocol-guarded—removal of any single component yields measurable, but non-catastrophic, degradation.

Implications, Future Directions, and Technical Impact

Agentic-SecPBFT establishes an effective model for integrating multi-agent reinforcement learning techniques into consensus protocol defense in realistic wireless MANETs. Its approach bridges a longstanding semantic gap between logical and physical identities, enabling proactive defensive policies that are channel-aware and context-sensitive. The framework's modularity and separation of decision advisories from protocol-critical commit logic enhance robustness to emergent adversarial strategies, topology churn, and physical capture.

Practically, Agentic-SecPBFT is an archetype for securing wireless distributed ledgers in vehicular, industrial, and edge AI deployments where mobility and environmental uncertainty fundamentally impair classical BFT assumptions. Theoretically, it motivates further research into Dec-POMDP hierarchy, probabilistic identity association, and cross-layer defense in decentralized networks.

Conclusion

Agentic-SecPBFT demonstrates that agentic AI-driven security architectures can substantially improve the integrity, resilience, and efficiency of wireless PBFT consensus in MANETs. Its hierarchical multi-agent MADQN algorithm, dynamic reputation management, and channel-aware defensive posture yield notable gains in attack detection, consensus reliability, latency, and throughput. The framework is broadly applicable to decentralized wireless systems requiring robust, adaptive security under adversarial and uncertain conditions (2607.03269).