Security Chaos Engineering

• Security Chaos Engineering is an emergent paradigm that applies chaos engineering principles to security by deliberately injecting faults or simulated attacks into systems.
• It uses hypothesis-driven experiments and multi-layer observability to expose hidden vulnerabilities and assess system resilience under realistic adverse conditions.
• Empirical studies indicate that SCE reduces incident impact and accelerates recovery through controlled perturbations, evidence-based refinements, and iterative testing.

Security Chaos Engineering (SCE) is an emergent paradigm in information security that applies principles of chaos engineering specifically to the security domain. It involves the deliberate injection of faults or simulated attacks into a live system or a production-like environment in order to study behavior under adverse conditions, expose hidden vulnerabilities, validate resilience to real-world threats, and support continuous improvement of system design and operation. In the literature surveyed here, SCE is closely associated with resilience over absolute prevention, hypothesis-driven experimentation, steady-state verification, blast-radius control, and multi-layer observability; it has been instantiated in containerized applications, JVM services, permissioned blockchains, cyber-physical systems, LLM-based multi-agent systems, DevSecOps workflows, and BAS platforms (Shortridge, 2023, Sánchez-Matas et al., 5 Aug 2025, Simonsson et al., 2019).

1. Conceptual foundations

A central premise of SCE is that attacks are inevitable, just as errors and faults are expected in complex systems. On that basis, the objective is not to assume that all attacks can be prevented, but to minimize the impact of attacks, generate actionable evidence, and use that evidence to refine architecture, controls, processes, and training. One account explicitly describes SCE as “Continuous Verification,” emphasizing continuous experimentation and system resilience rather than a binary notion of protection (Shortridge, 2023).

This orientation distinguishes SCE from approaches that justify security investment only by the absence of successful attacks. In the resilience-based framing, positive return is associated with reduced incident cost, faster response and recovery, less downtime, and increased business productivity. A representative formulation argues that SCE can preserve the same level of security incident impact even after doubling business revenue or increasing software delivery speed, and it treats time to response, containment, recovery, frequency of successful detection, and business productivity indicators as relevant metrics (Shortridge, 2023).

The same conceptual pattern appears outside conventional IT systems. In industrial cyber-physical systems, chaos engineering is presented as a resilience-enhancement methodology for systems that must remain available, secure, safe, and reliable against accidental and malicious adverse events. In that setting, malicious events are treated as part of the disturbance vector, and runtime CE-based analyses are proposed to predict environment changes and apply mitigation measures limiting the range and severity of the event and minimizing its blast radius (Konstantinou et al., 2021).

2. Experimental logic and methodological structure

Across the surveyed literature, SCE follows a recurring experimental cycle. One explicit formulation proceeds by defining hypotheses about how systems should respond to particular attack scenarios, injecting controlled faults or attacks that mirror realistic security incidents, observing and measuring the real-world responses of both systems and humans, analyzing deviations between expected and actual behavior, remediating and improving the system, and repeating the process regularly as a feedback loop for continuous improvement (Shortridge, 2023).

Steady state is a core construct in several domains. In cyber-physical systems, it is mathematically formalized through nonlinear state-space models such as

$\dot{x}(t) = g(x(t), u(t), \delta(t))$

and

$\Delta y(t) = h(x(t), u(t), \delta(t)),$

where malicious threats can be modeled as part of the disturbance input $\delta(t)$. In that setting, SCE-style experiments test the hypothesis that injected disturbances or attacks will not push the system outside steady-state bounds, and the analysis can be linked to equilibrium existence, exponential stability, and operational constraints (Konstantinou et al., 2021).

In containerized systems, the perturbation itself may be formalized at the syscall boundary. ChaosOrca specifies each perturbation as a tuple $(s, e, d)$, where $s$ is system call type, $e$ is the injected error code, and $d$ is the delay before invocation. The purpose is to assess an application’s self-protection capability with respect to system call errors while running under production-like workload and without instrumenting the application (Simonsson et al., 2019).

Recent SCE work for AI-intensive systems generalizes the same pattern. For LLM-based multi-agent systems, a proposed framework proceeds in three phases: literature and tooling review for identification of failure modes; framework development for modeling and injecting failures such as agent unavailability, message corruption or drop, and resource contention; and validation and iteration through case studies, action research, and auditing or certification of robustness. The framework emphasizes real-time monitoring of system state changes, performance metrics, error propagation, and behavioral divergence from expected operation (Owotogbe, 6 May 2025).

A closely related DevSecOps variant uses LLMs to automate attack-defense trees that can then be translated into SCE experiments. The flow is organized into six phases: application security context, prompt context, insert prompt, attack context, cosmetic context, and expert validation. This workflow is designed to make attack-defense tree generation more scalable and context-specific, while retaining expert scrutiny over inaccuracies or irrelevant attacks (Caballero et al., 15 Feb 2026).

3. Instrumentation, observability, and control surfaces

SCE depends on observability that spans multiple system layers. ChaosOrca provides a particularly explicit architecture with three components: Monitor, Perturbator, and Orchestrator. Monitoring is multi-layered: container-level CPU, RAM, network, and I/O statistics are collected via cAdvisor; system-call-level type, arguments, and results are captured via bpftrace; and application-level signals such as HTTP response codes and latencies are observed via PyShark. Data are stored in Prometheus and visualized in Grafana with overlays showing perturbation periods. The perturbation mechanism is implemented through strace and kernel-level syscall tracing or interception (Simonsson et al., 2019).

At the language-runtime level, ChaosMachine targets Java exception handling at the granularity of try-catch blocks. Its architecture consists of a Monitoring Sidecar, a Perturbation Injector, and a Chaos Controller. The system operates in observation mode, exploration mode, and falsification mode. Injection is performed through automated bytecode instrumentation using ASM, and only one injector is active at a time in order to minimize risk and simplify debugging. Default monitoring includes crash rates, HTTP status, log entries, and resource usage, while user-defined metrics can be added for domain-specific validation (Zhang et al., 2018).

In BAS-oriented SCE, observability and control are distributed across a three-layer architecture composed of an SCE Orchestrator layer, a Connector layer, and a BAS layer implemented with MITRE Caldera. The Orchestrator includes a Threat Intelligence Database, Chaos Experiment Designer, Attack Tree Generator, and ChaosXploit control logic. The Connector standardizes API interactions through modules such as State Fetcher, Operation Creator or Manager, Ability Executor, and Result Retriever. The BAS layer executes the actual adversary abilities, adversary profiles, campaigns, and plugins (Sánchez-Matas et al., 5 Aug 2025).

These systems share a common design principle: perturbation must be deliberate and reversible, and observability must be sufficient to detect not only overt crashes but also silent degradation, delayed recovery, unintended privilege escalation, and failures of alerting or logging. This suggests that, in practice, SCE is as much an observability discipline as an injection discipline.

4. Representative systems and empirical instantiations

In containerized applications, ChaosOrca was evaluated on TTorrent, Nginx, and Bookinfo under production-like workloads. The campaigns tested relevant syscalls with multiple error codes and delay values. Two major classes of issues were identified: non-resilient failure handling, in which syscall faults caused whole applications or critical services to crash, and resource overconsumption, in which syscall failures led to uncontrolled CPU or RAM consumption or network stalls. The critical syscalls and their effects varied by application, supporting the need for tailored rather than generic SCE (Simonsson et al., 2019).

In JVM systems, ChaosMachine was deployed on TTorrent, XWiki, and Broadleaf, covering 630k lines of code and 339 try-catch blocks. It classifies blocks as resilient, observable, debuggable, or silent. The evaluation reported 18 resilient try-catch blocks and 34 silent “debug nightmares,” thereby showing that resilience code itself can conceal serious operational and potentially security-relevant weaknesses. The same framework is described as extensible to security-specific exceptions such as failed authentication, authorization, or cryptographic operations (Zhang et al., 2018).

In permissioned blockchains, chaos engineering has been used to study PBFT on Hyperledger Sawtooth, Clique on Go Ethereum, and Raft on Hyperledger Fabric. The experiments used six validators, block size 10 transactions per block, Locust for load generation, Pumba for fault injection in Dockerized deployments, and the Linux tc tool for network faults in Geth. The primary metrics were write throughput, average write latency, and success rate. Baseline values reported in the study were 50 tx/s, 1100 ms, and 0.88 for PBFT at 250 users; 27.3 tx/s, 49 ms, and 1.0 for Clique at 250 users; and 5.8 tx/s, 1850 ms, and 0.98 for Raft at 50 users. Under chaos scenarios, PBFT was the most sensitive to faults, Clique was comparatively resilient to network faults but degraded when half the network was paused, and Raft showed generally poor performance and crash proneness in some scenarios (Sondhi et al., 2021).

In LLM-based multi-agent systems, chaos engineering has been proposed to surface vulnerabilities arising from hallucinations, agent failures, communication failures, emergent misbehavior, and cascading errors. The framework is intended for sandboxed testbeds and industrial-scale systems developed in collaboration with Deloitte, and it proposes both quantitative metrics such as response time, error rate, resource usage, and recovery time and qualitative metrics such as user experience, business impact, behavioral logs, and expert feedback. The emphasis is on robustness assessment and a potential certification path, although the detailed empirical results were described as forthcoming (Owotogbe, 6 May 2025).

In DevSecOps, an LLM-based flow has been used to generate attack-defense trees in DOT/Graphviz format for a military logistics system in AWS GovCloud using EC2, CodeBuild, and CodeGuru. The trees were evaluated with three metrics—attacks based on known TTPs, ordered attack procedures, and usable attack procedures—and an overall tree score. In the reported comparison, QwQ-32B scored 71.60% and GPT-4 scored 61.73%. A selected branch was then turned into an SCE experiment around privilege escalation via an EC2 Spot Instance, with AWS GuardDuty used for observability and a steady-state hypothesis that GuardDuty would detect and flag the incident (Caballero et al., 15 Feb 2026).

BAS integration extends this pattern by generating inferred attack trees from adversary profiles and executing them through MITRE Caldera. The objective is to move beyond static playbooks by trying multiple attack paths, including branches corresponding to different worms or lateral-movement techniques, and to collect richer reports on successful and failed branches (Sánchez-Matas et al., 5 Aug 2025).

5. Outcomes, value, and recurrent findings

The empirical literature repeatedly reports that SCE reveals weaknesses that are not easily exposed by conventional testing. In ChaosOrca, syscall-level perturbations uncovered crash-inducing error handling gaps and DoS-relevant resource overconsumption. In ChaosMachine, “silent” exception-handling paths showed that a system may fail without user-visible effects or useful logs. In blockchain experiments, platform architecture and protocol engineering materially affected outcomes, such that two platforms using the same consensus algorithm or protocol should not be assumed to have similar performance. This pattern suggests that SCE is particularly useful when failure semantics depend on cross-layer interactions rather than on isolated components (Simonsson et al., 2019, Zhang et al., 2018, Sondhi et al., 2021).

A second recurring finding is that SCE generates evidence that can be directly operationalized. One ROI-oriented argument holds that SCE produces valuable evidence to inform continuous improvement of system design and operation, and that the feedback loop from experiment to evidence to analysis to remediation can transform security investment from a wager on total prevention into a source of measurable resilience and productivity improvements. The same account cites empirical correlations in which standard chaos engineering is associated with a 2x likelihood of high resiliency, and SCE is associated with increased software velocity and less time spent remediating issues (Shortridge, 2023).

A third finding concerns adversary realism. In BAS-oriented and DevSecOps-oriented work, SCE is strengthened by explicit use of adversary profiles, MITRE ATT&CK TTPs, executable attack branches, and automated monitoring through tools such as MITRE Caldera and AWS GuardDuty. This narrows the gap between abstract threat modeling and concrete, repeatable experiments. A plausible implication is that SCE becomes more informative as the linkage between threat intelligence, attack-path generation, and observability becomes tighter (Sánchez-Matas et al., 5 Aug 2025, Caballero et al., 15 Feb 2026).

6. Limitations, misconceptions, and open directions

The current SCE literature also documents substantial limitations. ChaosOrca did not try all 100+ possible error codes per syscall and evaluated only three applications. ChaosMachine depends on workload coverage, observes primarily immediate effects on the current request, and can require manual analysis for semantic comparison of complex outputs. The LLM-MAS framework emphasizes planned rather than completed empirical validation, and the BAS architecture identifies mid-branch pivots and backtracking as future extensions. In DevSecOps tree generation, expert validation remains necessary to correct inaccuracies, irrelevant attacks, or missing links (Simonsson et al., 2019, Zhang et al., 2018, Owotogbe, 6 May 2025, Sánchez-Matas et al., 5 Aug 2025, Caballero et al., 15 Feb 2026).

A common misconception is to equate SCE with indiscriminate disruption. The surveyed systems instead stress controlled perturbation, sandboxed or semi-production execution, isolation through cgroups and namespaces, minimization of blast radius, one-injector-at-a-time policies, rollback mechanisms, and steady-state monitoring. The evidence base therefore frames SCE as disciplined experimentation rather than uncontrolled failure induction (Simonsson et al., 2019, Konstantinou et al., 2021, Sánchez-Matas et al., 5 Aug 2025).

There is also terminological heterogeneity. Some works discuss SCE directly, while others are described as aligned with SCE because they test resilience or self-protection under controlled faults. A related ambiguity appears in the record for “Evaluating Permissioned Blockchain Using Stochastic Modeling and Chaos Engineering,” whose provided content was reported to be a LaTeX template demo without substantive material on chaos engineering, blockchain consensus, or SCE. This indicates that bibliographic titles and abstracts alone may be insufficient for establishing evidentiary relevance, and that careful source validation remains necessary in this area (Sondhi et al., 2023).

Taken together, the literature portrays SCE as a hypothesis-driven, observability-intensive, and increasingly automation-oriented research program for probing security behavior under controlled adversity. Its practical expression varies by domain, but its recurrent structure—steady-state reasoning, perturbation design, multi-layer monitoring, evidence generation, and iterative hardening—remains consistent across contemporary systems research (Shortridge, 2023, Owotogbe, 6 May 2025).