---
title: 'FlyTrap: Automated PM Crash-Consistency Tester'
url: https://www.emergentmind.com/topics/flytrap
type: topic
---

# FlyTrap: Automated PM Crash-Consistency Tester

Searching arXiv for the specified FlyTrap paper and closely related persistent-memory crash-consistency context.
FlyTrap is a fully automated, record-and-replay framework for finding crash-consistency bugs in in-kernel persistent-memory (PM) file systems. It requires no source-level instrumentation, works on any POSIX-compliant PM file system, and exercises crashes both during and immediately after system calls. In the reported study, FlyTrap discovered 18 new bugs across four PM file systems; the bugs were confirmed by developers, and many had already been fixed. The reported consequences included breaking the atomicity of `rename()` and making the file system unmountable [2204.06066].

## 1. Scope and operating assumptions

FlyTrap is designed around a specific property of PM file systems: they bypass the block layer and use four centralized persistence primitives—non-temporal `memcpy`, non-temporal `memset`, cache-line flush, and store fence—to make data durable. Instead of instrumenting every store instruction, FlyTrap intercepts exactly those four functions via Linux Kprobes, logging each invocation’s arguments and return value, the identity of the current system call via entry and exit markers, and the contents of the flushed lines when a cache-line flush occurs [2204.06066].

The framework’s stated goal is: given a PM file-system implementation \(F\) and a workload \(W\), FlyTrap builds all “interesting” crash states arising during \(W\) and checks whether \(F\), upon recovery, lands in a correct state. The design is therefore persistence-centric rather than syscall-trace-centric in the conventional block-I/O sense. Because PM file systems expose durability through a small set of persistence primitives, FlyTrap can remain lightweight while still observing the writes that matter for crash consistency. The reported logging overhead is a few percent slowdown on typical workloads, aided by coalescing multiple writes to the same cache line [2204.06066].

A plausible implication is that FlyTrap treats PM crash consistency as a persistence-ordering and metadata-recovery problem at the granularity actually exercised by PM software, rather than as a replay of block requests or a source-instrumented semantic model.

## 2. Record-and-replay architecture and crash model

FlyTrap divides execution of each system call into epochs separated by persistence fences. Within an epoch, “in-flight writes” are writes to PM that have not yet been fenced. A crash can occur at any point—either just before a fence or after the system call returns—and leaves exactly the subset of in-flight writes that have already actually reached media. Because x86 enforces per-cache-line write ordering, FlyTrap does not need to permute writes; it only needs to consider all subsets \(S\) of in-flight writes. If \(P_i\) is the set of writes made durable up to fence \(i\) and \(W_i=\{w_{i1},\dots,w_{in}\}\) are the in-flight writes just before fencing, the generated crash states are [2204.06066]

$$
C_i = \{\, P_i \cup S \mid S \subseteq W_i \,\}.
$$

The end-to-end workflow has three phases. In the recording phase, the workload runs under a lightweight loadable kernel module that instruments the four persistence functions and emits a totally ordered log of syscall-entry markers, persistence calls, and syscall-exit markers. In replay, FlyTrap reconstructs a “live” PM image in user space, and at each fence and at the end of each system call it both emits the persisted image corresponding to a crash right after the epoch and enumerates all nonempty subsets of the current in-flight vector to model crashes in the middle of the system call. Normal forward progress is then modeled by committing all in-flight writes and clearing the in-flight set [2204.06066].

This crash model is notable because it is both exhaustive and bounded. Exhaustiveness comes from considering every subset of in-flight writes at each crash point, while boundedness comes from restricting attention to the persistence-level events actually observed during the workload. The framework therefore targets the combinatorial space induced by PM durability semantics, not the full space of architectural store interleavings.

## 3. Oracle construction and invariant checking

For each generated crash-state image \(C\), FlyTrap mounts the file system under test on \(C\). Failure to mount is itself reported as a bug. The checker then constructs two oracle images for the crashed syscall \(i\): \(pre_i\), the on-disk state just before the in-flight writes and fence of syscall \(i\), and \(post_i\), the state after completing syscall \(i\) and fencing. After recovery, the mounted state \(R\) must satisfy [2204.06066]

$$
R \in \{pre_i, post_i\}.
$$

For specific syscalls, FlyTrap also checks POSIX-implied invariants. The canonical example is `rename()` atomicity: if the workload renames `old→new`, recovery must satisfy

$$
(R \text{ contains new} \land \neg \text{contains(old)}) \;\lor\; (\neg \text{contains(new)} \land \text{contains(old)}).
$$

The checker is workload-aware but specification-light. It uses the simple pre-versus-post oracle to encode the requirement that each syscall is either fully applied or not applied at all, and it augments this with predicates over directory and inode metadata for POSIX guarantees such as link-count invariants and no lost allocations. FlyTrap automatically inspects the on-disk layout by enumerating inodes, directories, and data blocks to verify these predicates. To validate mount-time usability, it also attempts to create a new file in every directory and then delete it; any unexpected error or inconsistency is reported together with the original workload, crash point, and file-system version. Duplicate reports are suppressed by lexical and workload-based clustering [2204.06066].

A common misconception in crash testing is that a mountable image is therefore a correct image. FlyTrap’s design rejects that assumption explicitly: mount failure is only one class of error, while semantic violations of atomicity and metadata invariants remain first-class outcomes.

## 4. Evaluation on PM file systems

The evaluation covered PMFS, NOVA, NOVA-Fortis, and WineFS. Workloads came from two sources: ACE-based small workloads, including `seq-1`, `seq-2`, and the metadata-focused `seq-3`, and Syzkaller gray-box fuzzing workloads with unbounded length and coverage-driven generation. For each workload, FlyTrap exhaustively, up to a small cap on in-flight writes, generated crash states, mounted the file system, checked pre/post invariants, and reported discrepancies. All 18 unique bugs were reported upstream; developers confirmed 16 of them, and 12 had already been fixed [2204.06066].

The reported bug classes span both root cause and impact.

| Dimension | Category | Count |
|---|---|---:|
| Root cause | Logic errors in metadata or journaling code | 14 |
| Root cause | PM-programming errors | 4 |
| Optimization source | In-place metadata updates outside the journal | 6 |
| Optimization source | Recovery code that rebuilds volatile in-DRAM state | 7 |
| Optimization source | Complex resilience features in NOVA-Fortis | 5 |
| Impact | File system unmountable after crash | 3 |
| Impact | `rename()` atomicity violated | 2 |
| Impact | Data loss | 5 |
| Impact | Files or directories permanently inaccessible | 4 |

The examples given in the study are concrete. Among logic errors, bug `#4` is a NOVA `rename()` failure that drops a directory entry if a crash hits in-place unlink, and bug `#7` is a NOVA `truncate()` failure that loses data after unordered log replay. Among PM-programming errors, bug `#2` is a NOVA deferred flush issue on a new file, and bug `#10` is a NOVA-Fortis case leaving torn checksum blocks unreadable. These examples are important because they show that FlyTrap did not merely rediscover missing-flush defects; it exposed failures in journaling logic, recovery logic, and optimization paths [2204.06066].

## 5. Methodological lessons and design implications

**Logic and optimization dominate the bug landscape.** The study’s first stated lesson is that most PM-file-system crash-consistency bugs arise from higher-level logic or optimization mistakes—journal bypass, in-place updates, rebuild-on-mount—rather than from low-level flush or fence ordering. This suggests that testing strategies focused only on write atomicity or fence placement are incomplete for PM file systems [2204.06066].

**Crash points inside system calls are critical.** Eleven of the eighteen bugs only manifest if the crash occurs before the final persistence fence of a system call. A tool that crashes only at `fsync()` or `sync()` boundaries will therefore miss most PM-file-system bugs found in this study. That observation sharply distinguishes PM testing from many traditional crash-testing regimes [2204.06066].

**Small workloads can be disproportionately effective.** Fifteen of the eighteen bugs were exposed by ACE workloads of length \(\le 3\). This does not imply that fuzzing is unnecessary; Syzkaller remained part of the evaluation methodology. It does suggest that systematic bounded testing is a practical first line of defense even when PM implementations differ substantially from block-based file systems [2204.06066].

**Recovery-time code is a major fault surface.** Recovery code that rebuilds or validates in-DRAM metadata—free lists, per-CPU structures, and checksums—was identified as a significant source of bugs. This finding shifts attention from steady-state persistence paths to mount and replay logic, which is often less frequently exercised in routine testing [2204.06066].

Taken together, these lessons amount to a broader claim about PM systems methodology: crash consistency cannot be reduced to persistence primitives alone, even though FlyTrap observes those primitives. Correctness emerges only when persistence ordering, metadata mutation, journaling policy, and recovery reconstruction are validated as a coupled system.

## 6. Generality, extensions, and name ambiguity

FlyTrap’s architecture is described as general: any new in-kernel PM file system that exposes a small set of persistence primitives can be tested immediately by pointing FlyTrap at those functions. The core record/replay engine can be extended to handle concurrent workloads by logging per-thread execution order or by integrating with kernel tracing frameworks such as `ftrace` to capture interleavings. The same approach can also be adapted to user-space PM libraries such as PMDK by dynamically instrumenting their flush and fence APIs in a similar fashion. The code base, together with modified ACE and Syzkaller front ends, is available at `https://github.com/utsaslab/flytrap` [2204.06066].

The paper characterizes this combination of lightweight kprobe-based interception, exhaustive yet bounded crash-state exploration, and oracle-based consistency checks as the first off-the-shelf framework for systematically finding and analyzing crash-consistency bugs in persistent-memory file systems. This suggests that FlyTrap occupies a methodological position between heavyweight formal verification and ad hoc failure injection: it is automated enough for routine deployment, but precise enough to recover concrete crash states and invariants.

The name “FlyTrap” is not unique within recent arXiv literature. “FlyTrap: Physical Distance-Pulling Attack Towards Camera-based Autonomous Target Tracking Systems” denotes a physical-world attack framework that employs an adversarial umbrella, a progressive distance-pulling strategy, and closed-loop experiments on ATT drones [2509.20362]. Separately, “Virtual Lab by Quantum Flytrap” denotes a browser-based, no-code quantum-optics simulator associated with the Quantum Flytrap platform [2203.13300]. These uses are terminologically distinct from the PM crash-consistency framework, despite the shared name.

Source: https://www.emergentmind.com/topics/flytrap