File-Based Memory Systems

• File-based memory systems are architectures that map traditional in-memory data structures onto persistent storage using files, ensuring transparent persistence and high performance.
• They leverage hardware advances like byte-addressable persistent memory and innovative directory indexing techniques to boost lookup speeds and reduce write amplification.
• These systems support applications in high-performance computing, AI infrastructure, and context engineering by providing atomicity, crash consistency, and scalability.

A file-based memory system is a class of storage architecture in which data structures traditionally managed by volatile memory (RAM)—including arrays, hash tables, lists, and even application context—are mapped onto persistent storage, using files and file systems as the semantic and physical substrate. These systems provide transparent persistence, allow manipulation via familiar memory-like interfaces, and frequently leverage advances in hardware such as byte-addressable nonvolatile memory (NVM/PMem), memory-semantic SSDs, high-performance clusters, and near-data computation engines. The rigorous engineering of file-based memory systems involves rethinking path lookup, directory indexing, data consistency, write amplification, persistence mechanisms, and context traceability, and spans high-performance computing, embedded systems, AI infrastructure, and database workloads.

1. Directory Structures and Indexing Approaches

Efficient directory lookup is central to the performance of any file-based memory system. Traditional systems store dentries as variable-length arrays scanned linearly, incurring $O(n)$ lookup complexity. Persistent-memory oriented filesystems (PMFS, NOVA) compounded this problem with auxiliary in-memory B⁺-trees requiring reconstruction after reboot and imposing heavy write cost on splits and rebalances.

Content-Indexed Browsing (CIB) (Zhang et al., 2019) innovated by using the directory’s layout as its own on-disk index. Each directory is partitioned into fixed-size blocks, each with a bitmap marking active slots, and blocks are ordered by hash-key range. An optional contiguous array of block pointers allows binary search over block headers, yielding $O(\log B) + O(D) \approx O(\log n)$ lookup, with negligible PM write amplification. With block splits and atomic pointer updates, CIB maintains crash consistency without explicit logging. System-level benchmarks on CIB-PMFS demonstrated up to 176.1× path-lookup speedup and up to three orders of magnitude drop in PM writes compared to extra-DRAM B-Trees.

2. Byte-Addressable Persistent Memory and Dual-Mode I/O

Emerging persistent memory (PM)—such as Intel Optane DC—offers byte-addressability and sub-microsecond access latency, sharply reducing the bottleneck imposed by block-layer translation and page caching (Breukelen et al., 2023). File-based memory systems for PM often bypass the page cache, exposing mmap-style direct load/store access, eliminating cache pollution and I/O stalls inherent in block-based file systems (Dubeyko, 2019).

Systems like DurableFS (Kalita et al., 2018) employ copy-on-write for data blocks plus redo logging for metadata, committing all updates atomically at file-close. Write ordering and fence primitives are used to enforce crash consistency. KucoFS (Chen et al., 2019) further splits control/data by mapping the FS image into user space (initially read-only) and synchronizing fine-grained permission toggling via kernel threads, efficiently batching metadata updates for multi-core scalability while ensuring write protection at page granularity.

ByteFS (Li et al., 9 Jan 2025) generalized these strategies for CXL/PCIe-attached SSDs supporting both block and byte-granular I/O. Metadata structures were split to permit sub-block writes, SSD firmware used log-structured DRAM as a byte-write buffer, and coordinated host/SSD caching avoided double-buffering. ByteFS achieved up to 2.7× end-to-end speedup and 5.1× reduction in NAND flash traffic relative to Ext4/F2FS, and up to 25.3× cut in host-side metadata write amplification.

3. Consistency, Atomicity, and Crash-Safety Mechanisms

Guaranteeing per-operation atomicity, durability, and crash consistency—under hardware and software failures—is a hallmark of file-based memory system research. Strategies include:

• Log-Structured Metadata: Redo logs, as in DurableFS (Kalita et al., 2018), replay committed updates on recovery, guaranteeing "all-or-nothing" semantics at transactional boundaries (typically file close).
• Shadow Paging and Copy-on-Write: BPFS, PMFS, and derivatives (Breukelen et al., 2023) rely on atomic pointer swaps and copy-on-write for large updates, often combined with in-place atomic updates for small fields.
• Operation Ordering Enforcement: SquirrelFS (LeBlanc et al., 2024) uses Rust’s typestate to encode state-transition and ordering invariants between persistent structures at compile-time. Synchronous soft-updates enforce: never point to uninitialized structures, never reuse un-nullified resources, and never reset an old pointer before the new is fully durable. Compile errors automatically flag crash-inconsistent code paths, yielding provable safety with no runtime penalty. SquirrelFS matches or surpasses legacy journal-based systems in throughput, with zero runtime overhead from the typestate checks.
• Consistency Modes: SplitFS (Kadekodi et al., 2019) separates data and metadata planes, allowing applications to select among POSIX, sync, and strict consistency models. The strict mode combines per-process operation logs and atomic extent swaps (relink()), delivering full atomicity on both data and metadata with minimal overhead.

4. Performance, Write Amplification, and Wear-Leveling

File-based memory systems mitigate performance bottlenecks stemming from write amplification and block/erase asymmetry in flash, SSDs, and PMem.

• NAND Flash File Systems: JFFS2, YAFFS2, and UBIFS (Olivier et al., 2012) employ log-structured out-of-place updates, garbage collection for invalid pages, and wear-leveling to distribute erase counts. YAFFS2 uses checkpointing for fast mount and O(1) lookup, while UBIFS uses transactional B-trees and explicit LEB properties tree for wear-level selection. Write amplification (WA) is defined as $$(\text{Bytes written to flash})/(\text{Bytes written by host})$$, and efficient designs sustain WA $\approx 1.1$–$1.3$ (UBIFS), with better compression and batch GC.
• Roomy (Disk-based Memory System): Roomy (Kunkle, 2010) dynamically stripes and buckets file-based structures across disks in a cluster. Delayed operations and bulk synchronization minimize random I/O, and network-transparent APIs hide MPI/disk mechanics. Aggregate bandwidth scales as $D \cdot B$ disks and $P \cdot D \cdot B$ in a cluster; synchronization cost is amortized by sorting and per-bucket streaming.
• Persistent Memory Systems: PMFS, NOVA, SplitFS, KucoFS, SquirrelFS (Breukelen et al., 2023, Kadekodi et al., 2019, Chen et al., 2019, LeBlanc et al., 2024) all achieve low-latency small-I/O throughput by leveraging direct-access, minimal fencing, lock-free concurrency, and metadata offload. Table-based performance comparison (normalized to ext4-DAX) demonstrates small-update latency improvement up to $3.6\times$ (ctFS) and small-I/O throughput up to $2.5\times$ (HashFS), with index-offload architectures dominating scalability.

5. Memory Systems for Context Engineering and Generative AI

Contemporary research extends file-based memory system principles beyond storage, to human/machine context engineering pipelines (Xu et al., 5 Dec 2025). The Agentic File System (AFS), implemented in AIGNE, mounts heterogeneous memory, knowledge, tool, and annotation backends as files under a common namespace.

• Context-Engineering Pipeline: The Constructor selects context to fit within a token budget using metadata such as tokenCount and recency; the Loader materializes context into LLM prompt buffers; the Evaluator checks output for semantic consistency, then writes persistent memory files or routes for human review. All transactions are versioned, ACL-protected, and logged for auditability.
• Governance and Traceability: File-based persistence and explicit directory trees ensure that every context artifact—episodic memory, facts, toolkit descriptors, review logs—is auditable and governed by access control policies, supporting reproducible, accountable GenAI workflows.

6. Application Domains and Representative Use Cases

File-based memory systems underpin a range of application domains:

System or Approach	Target Storage	Use Case
CIB (PMFS) (Zhang et al., 2019)	NVRAM / PMem	Fast directory lookup in PM filesystems
ByteFS (Li et al., 9 Jan 2025)	NAND/CXL SSD	Unified byte/block I/O for SSD firmware
Roomy (Kunkle, 2010)	Clustered Disks	Graph BFS, algebra, large memory-bound computation
KucoFS (Chen et al., 2019)	Optane DC PM	Scalable, zero-syscall user-space access
Abacus (Garrison et al., 2021)	POSIX Shared Memory	Large-scale simulation checkpointing
AFS/AIGNE (Xu et al., 5 Dec 2025)	Agent context persistent stores	Context management in GenAI pipelines

These systems demonstrate that file-based memory abstractions, when mapped onto appropriate backends (PMem, SSD, RAMDISK, distributed disk), can achieve near-memory performance, scalability across hundreds of cores/nodes, transactional persistence, and traceability—all with minimal changes to application-level code.

7. Future Directions, Open Problems, and Limitations

Challenges remain in generalizing file-based memory systems across hardware platforms and software semantics:

• Unified OS Abstraction: Existing hybrid architectures often bypass or heavily modify the VFS. There is ongoing effort to unify kernel support for byte-addressable PMem, atomic in-place writes, and hardware page-table mapping under a single abstraction (Breukelen et al., 2023).
• Multi-Node, Multi-Tier Persistence: As PM is integrated into coherent fabrics (CXL/RDMA), file-based memory systems must adapt for distributed persistence, cross-device crash-recovery, and fine-grained access control.
• POSIX Compatibility vs. Performance: Index-offload and direct-access designs may diverge from strict POSIX file semantics, which raises adoption concerns; a compromise abstraction that balances programmability with hardware efficiency remains a point of investigation.
• Crash-Consistency Verification: Compile-time enforcement of consistency invariants (SquirrelFS (LeBlanc et al., 2024)) is promising but currently limited to ordering properties and types; dynamic set properties require richer type systems and runtime verification for full coverage.
• Write Amplification and Endurance: Efficient metadata and data layout, wear-leveling, and GC strategies are critical for minimizing write amplification and maximizing device lifetime, especially in flash and SSD scenarios (Olivier et al., 2012, Li et al., 9 Jan 2025).

File-based memory systems continue to be refined to exploit new hardware capabilities, support scalable, persistent, and auditable memory-backed computation, and converge memory/storage abstractions for next-generation workloads.