FlyTrap: Automated PM Crash-Consistency Tester

• FlyTrap is a fully automated record-and-replay framework that detects crash-consistency bugs in in-kernel persistent-memory file systems by intercepting key persistence primitives.
• It leverages Linux Kprobes to log non-temporal memcpy, memset, cache-line flushes, and store fences, achieving exhaustive exploration of crash states with minimal overhead.
• Evaluations on PMFS, NOVA, and related systems revealed 18 unique bugs, emphasizing its effectiveness in uncovering logic errors and optimization flaws in crash recovery.

Searching arXiv for the specified FlyTrap paper and closely related persistent-memory crash-consistency context. FlyTrap is a fully automated, record-and-replay framework for finding crash-consistency bugs in in-kernel persistent-memory (PM) file systems. It requires no source-level instrumentation, works on any POSIX-compliant PM file system, and exercises crashes both during and immediately after system calls. In the reported study, FlyTrap discovered 18 new bugs across four PM file systems; the bugs were confirmed by developers, and many had already been fixed. The reported consequences included breaking the atomicity of rename() and making the file system unmountable (LeBlanc et al., 2022).

1. Scope and operating assumptions

FlyTrap is designed around a specific property of PM file systems: they bypass the block layer and use four centralized persistence primitives—non-temporal memcpy, non-temporal memset, cache-line flush, and store fence—to make data durable. Instead of instrumenting every store instruction, FlyTrap intercepts exactly those four functions via Linux Kprobes, logging each invocation’s arguments and return value, the identity of the current system call via entry and exit markers, and the contents of the flushed lines when a cache-line flush occurs (LeBlanc et al., 2022).

The framework’s stated goal is: given a PM file-system implementation $F$ and a workload $W$, FlyTrap builds all “interesting” crash states arising during $W$ and checks whether $F$, upon recovery, lands in a correct state. The design is therefore persistence-centric rather than syscall-trace-centric in the conventional block-I/O sense. Because PM file systems expose durability through a small set of persistence primitives, FlyTrap can remain lightweight while still observing the writes that matter for crash consistency. The reported logging overhead is a few percent slowdown on typical workloads, aided by coalescing multiple writes to the same cache line (LeBlanc et al., 2022).

A plausible implication is that FlyTrap treats PM crash consistency as a persistence-ordering and metadata-recovery problem at the granularity actually exercised by PM software, rather than as a replay of block requests or a source-instrumented semantic model.

2. Record-and-replay architecture and crash model

FlyTrap divides execution of each system call into epochs separated by persistence fences. Within an epoch, “in-flight writes” are writes to PM that have not yet been fenced. A crash can occur at any point—either just before a fence or after the system call returns—and leaves exactly the subset of in-flight writes that have already actually reached media. Because x86 enforces per-cache-line write ordering, FlyTrap does not need to permute writes; it only needs to consider all subsets $S$ of in-flight writes. If $P_i$ is the set of writes made durable up to fence $i$ and $W_i=\{w_{i1},\dots,w_{in}\}$ are the in-flight writes just before fencing, the generated crash states are (LeBlanc et al., 2022)

$C_i = \{\, P_i \cup S \mid S \subseteq W_i \,\}.$

The end-to-end workflow has three phases. In the recording phase, the workload runs under a lightweight loadable kernel module that instruments the four persistence functions and emits a totally ordered log of syscall-entry markers, persistence calls, and syscall-exit markers. In replay, FlyTrap reconstructs a “live” PM image in user space, and at each fence and at the end of each system call it both emits the persisted image corresponding to a crash right after the epoch and enumerates all nonempty subsets of the current in-flight vector to model crashes in the middle of the system call. Normal forward progress is then modeled by committing all in-flight writes and clearing the in-flight set (LeBlanc et al., 2022).

This crash model is notable because it is both exhaustive and bounded. Exhaustiveness comes from considering every subset of in-flight writes at each crash point, while boundedness comes from restricting attention to the persistence-level events actually observed during the workload. The framework therefore targets the combinatorial space induced by PM durability semantics, not the full space of architectural store interleavings.

3. Oracle construction and invariant checking

For each generated crash-state image $C$, FlyTrap mounts the file system under test on $W$0. Failure to mount is itself reported as a bug. The checker then constructs two oracle images for the crashed syscall $W$1: $W$2, the on-disk state just before the in-flight writes and fence of syscall $W$3, and $W$4, the state after completing syscall $W$5 and fencing. After recovery, the mounted state $W$6 must satisfy (LeBlanc et al., 2022)

$W$7

For specific syscalls, FlyTrap also checks POSIX-implied invariants. The canonical example is rename() atomicity: if the workload renames old→new, recovery must satisfy

$W$8

The checker is workload-aware but specification-light. It uses the simple pre-versus-post oracle to encode the requirement that each syscall is either fully applied or not applied at all, and it augments this with predicates over directory and inode metadata for POSIX guarantees such as link-count invariants and no lost allocations. FlyTrap automatically inspects the on-disk layout by enumerating inodes, directories, and data blocks to verify these predicates. To validate mount-time usability, it also attempts to create a new file in every directory and then delete it; any unexpected error or inconsistency is reported together with the original workload, crash point, and file-system version. Duplicate reports are suppressed by lexical and workload-based clustering (LeBlanc et al., 2022).

A common misconception in crash testing is that a mountable image is therefore a correct image. FlyTrap’s design rejects that assumption explicitly: mount failure is only one class of error, while semantic violations of atomicity and metadata invariants remain first-class outcomes.

4. Evaluation on PM file systems

The evaluation covered PMFS, NOVA, NOVA-Fortis, and WineFS. Workloads came from two sources: ACE-based small workloads, including seq-1, seq-2, and the metadata-focused seq-3, and Syzkaller gray-box fuzzing workloads with unbounded length and coverage-driven generation. For each workload, FlyTrap exhaustively, up to a small cap on in-flight writes, generated crash states, mounted the file system, checked pre/post invariants, and reported discrepancies. All 18 unique bugs were reported upstream; developers confirmed 16 of them, and 12 had already been fixed (LeBlanc et al., 2022).

The reported bug classes span both root cause and impact.

Dimension	Category	Count
Root cause	Logic errors in metadata or journaling code	14
Root cause	PM-programming errors	4
Optimization source	In-place metadata updates outside the journal	6
Optimization source	Recovery code that rebuilds volatile in-DRAM state	7
Optimization source	Complex resilience features in NOVA-Fortis	5
Impact	File system unmountable after crash	3
Impact	rename() atomicity violated	2
Impact	Data loss	5
Impact	Files or directories permanently inaccessible	4

The examples given in the study are concrete. Among logic errors, bug #4 is a NOVA rename() failure that drops a directory entry if a crash hits in-place unlink, and bug #7 is a NOVA truncate() failure that loses data after unordered log replay. Among PM-programming errors, bug #2 is a NOVA deferred flush issue on a new file, and bug #10 is a NOVA-Fortis case leaving torn checksum blocks unreadable. These examples are important because they show that FlyTrap did not merely rediscover missing-flush defects; it exposed failures in journaling logic, recovery logic, and optimization paths (LeBlanc et al., 2022).

5. Methodological lessons and design implications

Logic and optimization dominate the bug landscape. The study’s first stated lesson is that most PM-file-system crash-consistency bugs arise from higher-level logic or optimization mistakes—journal bypass, in-place updates, rebuild-on-mount—rather than from low-level flush or fence ordering. This suggests that testing strategies focused only on write atomicity or fence placement are incomplete for PM file systems (LeBlanc et al., 2022).

Crash points inside system calls are critical. Eleven of the eighteen bugs only manifest if the crash occurs before the final persistence fence of a system call. A tool that crashes only at fsync() or sync() boundaries will therefore miss most PM-file-system bugs found in this study. That observation sharply distinguishes PM testing from many traditional crash-testing regimes (LeBlanc et al., 2022).

Small workloads can be disproportionately effective. Fifteen of the eighteen bugs were exposed by ACE workloads of length $W$9. This does not imply that fuzzing is unnecessary; Syzkaller remained part of the evaluation methodology. It does suggest that systematic bounded testing is a practical first line of defense even when PM implementations differ substantially from block-based file systems (LeBlanc et al., 2022).

Recovery-time code is a major fault surface. Recovery code that rebuilds or validates in-DRAM metadata—free lists, per-CPU structures, and checksums—was identified as a significant source of bugs. This finding shifts attention from steady-state persistence paths to mount and replay logic, which is often less frequently exercised in routine testing (LeBlanc et al., 2022).

Taken together, these lessons amount to a broader claim about PM systems methodology: crash consistency cannot be reduced to persistence primitives alone, even though FlyTrap observes those primitives. Correctness emerges only when persistence ordering, metadata mutation, journaling policy, and recovery reconstruction are validated as a coupled system.

6. Generality, extensions, and name ambiguity

FlyTrap’s architecture is described as general: any new in-kernel PM file system that exposes a small set of persistence primitives can be tested immediately by pointing FlyTrap at those functions. The core record/replay engine can be extended to handle concurrent workloads by logging per-thread execution order or by integrating with kernel tracing frameworks such as ftrace to capture interleavings. The same approach can also be adapted to user-space PM libraries such as PMDK by dynamically instrumenting their flush and fence APIs in a similar fashion. The code base, together with modified ACE and Syzkaller front ends, is available at https://github.com/utsaslab/flytrap (LeBlanc et al., 2022).

The paper characterizes this combination of lightweight kprobe-based interception, exhaustive yet bounded crash-state exploration, and oracle-based consistency checks as the first off-the-shelf framework for systematically finding and analyzing crash-consistency bugs in persistent-memory file systems. This suggests that FlyTrap occupies a methodological position between heavyweight formal verification and ad hoc failure injection: it is automated enough for routine deployment, but precise enough to recover concrete crash states and invariants.

The name “FlyTrap” is not unique within recent arXiv literature. “FlyTrap: Physical Distance-Pulling Attack Towards Camera-based Autonomous Target Tracking Systems” denotes a physical-world attack framework that employs an adversarial umbrella, a progressive distance-pulling strategy, and closed-loop experiments on ATT drones (Xie et al., 24 Sep 2025). Separately, “Virtual Lab by Quantum Flytrap” denotes a browser-based, no-code quantum-optics simulator associated with the Quantum Flytrap platform (Migdał et al., 2022). These uses are terminologically distinct from the PM crash-consistency framework, despite the shared name.