CSnake: Fault Injection in Distributed Systems

• CSnake is a fault injection framework that uses causal stitching to link sequential fault injections and simulate complex failure propagation in distributed systems.
• It applies counterfactual fault causality analysis by comparing injected runs with fault-free profiles to identify causal relationships triggering cascading errors.
• A three-phase allocation protocol efficiently navigates vast fault and workload combinations to expose previously undetected self-sustaining failure cycles.

CSnake is a fault injection framework to expose self-sustaining cascading failures in distributed systems. It uses the idea of causal stitching, which causally links multiple single-fault injections in different tests to simulate complex fault propagation chains. To identify these chains, it designs a counterfactual causality analysis of fault propagations—fault causality analysis (FCA)—that compares the execution trace of a fault injection run with its corresponding profile run and identifies any additional faults triggered, which are considered to have a causal relationship with the injected fault. To address the large search space of fault and workload combinations, CSnake employs a three-phase allocation protocol of test budget and performs a local compatibility check before connecting fault propagations from different workloads (Qian et al., 30 Sep 2025).

1. Problem setting and motivation

Self-sustaining cascading failures (“self-sus”) occur when a primary fault in a distributed system propagates through a sequence of components, each step activated by distinct conditions, and eventually reinjects the original fault, creating a closed loop that overwhelms built-in fault tolerance. Such failures are challenging to catch prior to deployment because no single test or fault injection typically exercises all the required triggering conditions in one shot.

The difficulty follows from the structure of cascading failures themselves. A system evolves through states “Correct” $\to$ (inject fault $f$) $\to$ “Error” $\to$ possibly masked by recovery $\to$ (under special condition) propagate to new error $\to \cdots \to$ eventual failure. In this setting, a “condition” is a logical predicate over the system state required to activate a particular propagation.

Existing fault-injection tools either inject coarse-grained external faults, such as node crashes and partitions, or rely on manually crafted or benchmark workloads that lack the nuanced combinations of conditions. CSnake addresses this gap by injecting single faults—exceptions, delays, and negations—into many existing integration tests, discovering one-step fault propagations, and reconstructing longer chains across tests. This suggests a decomposition strategy for failures that are otherwise not observable in a single execution.

2. Causal stitching and fault causality analysis

At the conceptual level, CSnake treats a causal relationship $f_1 \to f_2$ discovered in test $t$ as a length-1 propagation chain from $f_1$ to $f_2$. Causal stitching then links $f$0 observed in $f$1 with $f$2 observed in $f$3 to form $f$4, provided that $f$5 and $f$6 are compatible. Rather than attempt to trigger the entire cycle in one test, the framework breaks it into steps: each fault injection uncovers one step’s propagation under weaker, per-step conditions.

The causality notion is explicitly counterfactual. $f$7 is a counterfactual cause of $f$8 iff $f$9 would not occur unless $\to$0 occurs. The paper characterizes this as the strongest form of causality on Pearl’s “ladder.” FCA operationalizes this definition by comparing the execution trace of a fault injection run with its corresponding profile run, that is, the same test without the injection.

The injected fault types are exception injection, including system-specific or library-declared exceptions; delay injection, described as “contention” simulation, into loop iterations with 7 progressively larger delays from 100 ms to 8 s; and negation injection, which flips the boolean return of system-specific error detectors. For each injection run of fault $\to$1 in test $\to$2, the framework records which exception or negation points were encountered, termed execution trace interference, and for each loop, the total iteration counts, which are compared via a one-sided $\to$3-test at $\to$4 to detect statistically significant increases. Both the injection run and the fault-free profile run are repeated 5 times to reduce nondeterminism (Qian et al., 30 Sep 2025).

The additional faults triggered by injecting $\to$5 are represented as

$\to$6

These additional faults may be exceptions, delays in other loops, or negations. The framework also records two special interference types for nested or consecutive loops: if loop $\to$7 is nested under $\to$8, an injection into $\to$9 that slows $\to$0 may also slow $\to$1; similarly, delay may propagate to the next loop for consecutive loops.

The summary enumerates six interference types: delay $\to$2 exception/negation $\to$3 through trace interference; delay $\to$4 delay $\to$5 in another loop through iteration interference; exception/negation $\to$6 exception/negation $\to$7; exception/negation $\to$8 delay $\to$9; nested-loop propagation; and sibling/consecutive-loop propagation. In aggregate, these interference relations form the edge set that later supports chain construction and cycle detection.

3. Three-phase allocation protocol

A central systems problem for CSnake is the scale of the search space. The goal is to explore tens of millions of $\to$0 combinations under a limited budget of approximately $\to$1 runs, while maximizing discovery of distinct causal relationships, especially conditional ones. The framework therefore uses a three-phase allocation protocol (3PA).

The first principle is causally equivalent faults: faults that produce nearly the same interference set $\to$2 are clustered so that runs are not wasted by injecting them into the same test. The second principle is conditional causal relationships: if $\to$3 causes $\to$4 in some tests but $\to$5 in others, more injections of $\to$6 should be prioritized to extend both branches.

In Phase 1, causally equivalent fault detection consumes 25% of the budget. For each fault $\to$7, the framework picks the single test $\to$8 that maximizes code coverage over $\to$9’s location, computes $\to \cdots \to$0, and vectorizes the result via IDF:

$\to \cdots \to$1

where $\to \cdots \to$2 is the total injection experiments and $\to \cdots \to$3 is the number of times $\to \cdots \to$4 appears as interference. Then

$\to \cdots \to$5

with

$\to \cdots \to$6

followed by $\to \cdots \to$7 normalization. Hierarchical clustering with cosine distance groups faults into clusters $\to \cdots \to$8 of causally equivalent faults.

In Phase 2, causality exploration uses 50% of the budget. The algorithm proceeds round-robin across clusters $\to \cdots \to$9; when a cluster’s turn arrives, it randomly picks one fault $f_1 \to f_2$0 and injects it into one new test not seen so far. New $f_1 \to f_2$1 results are collected and re-vectorized, and a second IDF model is trained. For each cluster $f_1 \to f_2$2, the framework computes an intra-cluster similarity score

$f_1 \to f_2$3

Lower SimScore indicates more diverse interference and therefore more conditional causal relations.

In Phase 3, conditional-causality-guided extension consumes the remaining 25% of the budget. Each cluster $f_1 \to f_2$4 is assigned weight

$f_1 \to f_2$5

with $f_1 \to f_2$6, so that clusters with lower SimScore receive higher chance of extra injections. Unused quotas from exhausted clusters are transferred to under-explored ones guided by $f_1 \to f_2$7. A plausible implication is that 3PA is not merely a sampling heuristic but an explicit mechanism for biasing the exploration budget toward faults whose effects vary across workloads (Qian et al., 30 Sep 2025).

4. Compatibility analysis and implementation architecture

Stitching two causal edges requires more than matching endpoint faults. Before connecting $f_1 \to f_2$8 observed in test $f_1 \to f_2$9 and $t$0 observed in test $t$1, the framework must ensure that the conditions that triggered $t$2 in $t$3 are compatible with the conditions for $t$4 in $t$5. The ideal formulation is to collect full symbolic path conditions $t$6 and $t$7 at the propagation points and check $t$8.

The implemented solution is an approximate low-overhead local compatibility check. First, the framework records the two nearest callers of the fault location, excluding the location’s own method; both edges must have the same 2-level context for $t$9. Second, inside the loop or function enclosing $f_1$0, it records each encountered branch and its true/false outcome, using all if-conditions in that method or loop as monitor points. These local traces must match exactly across the two tests for the iteration or invocation where $f_1$1 occurred. The example given is a fault thrown inside createTmp(), called by BlockReceiver(): both runs must have the call stack …→BlockReceiver()→createTmp()→(fault), and all boolean decisions in createTmp()’s body leading to the fault must match.

The implementation architecture consists of a static analyzer, a runtime agent, a test runner, and an offline data pipeline. The static analyzer uses WALA to identify throw-points for exception injection, loops for delay injection, and boolean functions for negation injection, and instruments hooks via a custom Byteman agent. The runtime agent delivers a fault injection and monitoring hook whenever a configured injection point is reached. The test runner is a modified JUnit that discovers only parameter generation for parameterized tests, forks and CRIU-restores JVM snapshots, one per test, to speed up initialization, and runs each profile and injection 5 times. The data pipeline is described in four stages: the test driver picks $f_1$2 according to 3PA; the runtime agent injects and logs interfered faults and traces; offline FCA compares traces to produce edges $f_1$3; and the bug detector performs compatibility checks and beam search to find cycles (Qian et al., 30 Sep 2025).

5. Experimental evaluation

The evaluation uses the latest stable versions of HDFS 2.10.2, HDFS 3.4.1, HBase 2.6.0, OZone 1.4.0, and Flink 1.20.0. Experiments run on two Ubuntu 22.04 servers, one Xeon Gold with 512 GB and one EPYC with 256 GB, using Docker containers with 6 cores, a 32 GB heap, and Java 1.8. The beam size is $f_1$4 million chains.

Across the five systems, the paper reports the following ranges of static injection points: loops, 1,361–3,227; exceptions, 2,316–2,707; negations, 395–1,002; branch monitors, 18 212–38 192; and integration tests, 1 219–4 436. These figures indicate that the candidate space is large even before composing faults with workloads.

The main empirical result is that CSnake detected 15 previously unknown self-sus across the five systems; 5 were confirmed and 2 were already fixed. The cycles involve on average 1 delay plus 1–3 exceptions or negations. The “Alloc.” column in the evaluation indicates in which 3PA phase all required edges were discovered. Under the same number of runs, 4 of the 15 would be missed by random budget allocation, and 11 of the 15 cannot be triggered in any single test under a naïve single-fault-per-test strategy.

Two case studies illustrate the target phenomenon. In the HBase region-assignment retry case, heavy write plus region-assignment IOE causes an RS to be excluded from balancer, the balancer fails, an infinite retry loop follows, and the load increases; the resulting cycle is $f_1$5, and it is triggered only by stitching across three tests. In the HDFS IBR throttling bypass case, delay in large IBR causes RPC exception in IBR, and a test of IBR interval config shows immediate IBR retry, producing more IBRs without throttling.

The evaluation also studies cycle clustering and false positives. Without a limit on the number of delays, reported cycles include HDFS 2: 38 $f_1$6 15 clusters (6 TP), HBase: 72 $f_1$7 24 clusters (2 TP), and Flink: 48 $f_1$8 35 clusters (2 TP). Restricting beam search to at most one delay dramatically reduces false positives while retaining all true positives. The reported average latency overhead for profile runs is +185%, with a range of 63–376%, dominated by branch and stack tracing; the paper states that production is not the target, so no sampling or hardware tracing optimization is applied. For adoption effort, a newcomer can set up Sna on a new JVM-based system in approximately 0.5 day, including compile and test environment (Qian et al., 30 Sep 2025).

6. Limitations, false positives, and practical implications

The paper identifies several limitations. False positives arise mainly from uninteresting performance interferences, such as throttling storms, or loops that expand iteration counts but are already managed by adaptive backoff. Manual triage of cycle clusters is therefore required. The local compatibility check is approximate; in theory contradictory conditions could still slip through, although none were observed in practice.

The stated future directions are to use symbolic constraint collection and path conditions for stronger compatibility checks, to apply sampling or Intel-PT tracing to reduce overhead, and to refine 3PA for regression by biasing toward code areas with recent changes or past bugs. These points locate CSnake within a broader tradeoff between causality precision, runtime overhead, and search efficiency.

The lessons for practitioners are operational rather than algorithmic. The paper recommends separating heavyweight work, such as block reports, from critical paths such as heartbeats; implementing proactive throttling with client feedback to avoid retry storms; and prioritizing critical requests in asynchronous queues. A plausible implication is that the value of CSnake is not limited to bug detection: the recovered cycles expose architectural pressure points where fault tolerance and scheduling interact in ways that can sustain failure rather than absorb it (Qian et al., 30 Sep 2025).