Half a Century of Distributed Byzantine Fault-Tolerant Consensus: Design Principles and Evolutionary Pathways

Abstract: The concept of distributed consensus originated in the 1970s and gained widespread attention following Leslie Lamport's influential publication on the Byzantine Generals Problem in the 1980s. Over the past five decades, distributed consensus has become an extensively researched field. Practical Byzantine Fault Tolerance (PBFT) has emerged as a prominent and widely adopted solution due to its conceptual clarity, effectiveness, and resilience to arbitrary failures. However, PBFT does not universally address all scenarios, highlighting the necessity of developing a comprehensive understanding of the history, evolution, and foundational principles of distributed consensus. This article systematically reviews the historical evolution and foundational principles of distributed consensus, examining pivotal advancements including fault-tolerant state machine replication (SMR), consensus protocols in partially synchronous and asynchronous networks, and recent innovations in Directed Acyclic Graph (DAG)-based consensus mechanisms. We further analyse the core design rationales, essential components, and underlying primitives across various distributed fault-tolerant protocols. The relationship between BFT consensus mechanisms and their applications in environments requiring robust resilience against adversarial faults is also explored. Finally, we discuss emerging research areas and challenges, such as consensus for wireless and blockchain scenarios, highlighting potential future developments. This comprehensive overview offers valuable insights to inform the design, optimisation, and implementation of distributed consensus systems across multiple application scenarios.