---
title: 'DaxFS: CXL Shared Memory Filesystem'
url: https://www.emergentmind.com/topics/daxfs
type: topic
---

# DaxFS: CXL Shared Memory Filesystem

Searching arXiv for the primary DaxFS paper and closely related DAX filesystem context papers.
DaxFS is a Linux filesystem for CXL-based, byte-addressable disaggregated memory whose defining property is that all cross-host coordination is expressed through coherent 64-bit `cmpxchg` operations on shared memory. It is designed for CXL 3.0 Global Fabric Attached Memory (GFAM), where multiple hosts can issue load/store operations and fabric-wide atomics to the same physical memory region, and it uses that capability to provide lock-free multi-host writes, a cooperative shared page cache, and zero-copy pointer-based access without a centralized coordinator or journaling in the hot path [2604.01620].

## 1. Context, design target, and relation to DAX

DaxFS is situated at the intersection of DAX-style direct access and CXL-coherent disaggregated memory. CXL 3.0 introduces GFAM, in which memory devices on a CXL switch fabric can be accessed by multiple independent hosts as byte-addressable memory, with hardware cache coherence maintained by a device coherence engine and with 64-bit `cmpxchg` atomicity preserved across host boundaries. DaxFS targets precisely that environment: a shared-memory substrate in which filesystem data and metadata can be manipulated via ordinary loads, stores, and atomics rather than RPCs or block I/O [2604.01620].

The design begins from the observation that existing filesystems fit this environment poorly. Per-host DAX filesystems such as `ext4-dax`, NOVA, PMFS, and `tmpfs` preserve separate per-host metadata and caching state even when backed by the same physical memory, thereby duplicating caches and negating memory pooling. Distributed filesystems such as NFS, CephFS, Lustre, Orion, and Assise rely on message passing and server-mediated coordination, introducing microsecond-scale protocol costs before accesses that could otherwise be nanosecond-scale loads and stores. FamFS allows multi-host mounting of CXL-backed storage, but it uses a single-master model in which only one host can create, write, or delete while others replay metadata logs [2604.01620].

In broader DAX literature, direct access is usually framed as bypassing the page cache so that file contents in persistent memory are reached through direct mappings and ordinary load/store instructions. That reduces software overhead, but it does not by itself depart from a CPU-centric execution model; computation still occurs on host processors even when data is byte-addressable and page-cache bypassed [1901.01340]. DaxFS differs by treating shared coherent memory itself as the filesystem substrate for concurrent multi-host coordination.

A common misconception is that DaxFS is merely another DAX-enabled filesystem. More precisely, it is a CXL-native shared filesystem whose design premise is that cross-host coordination can be reduced to a single primitive—CAS—because the interconnect itself supplies coherent atomics. This suggests a distinct filesystem design point: shared memory is treated as storage, not merely as a faster backing medium for an otherwise conventional filesystem.

## 2. Architectural organization and filesystem modes

DaxFS is implemented as a Linux filesystem module of approximately 2.5 KLoC and is exposed as filesystem type `daxfs`. It uses the Linux VFS, but all persistent filesystem structures are stored directly in a DAX-mapped shared region, either by mapping a physical memory range with `memremap()` or by using `dma-buf` mappings for GPU-aware paths. The filesystem supports `mount(2)` and `fsopen/fsconfig/fsmount`, and it uses DAX-style mappings without going through `dax_device`; instead it maps the underlying physical memory via `memremap(MEMREMAP_WB)` or `dma_buf_vmap()` [2604.01620].

Within the shared region, DaxFS organizes data into a small number of explicit regions: a 4 KB superblock; an optional read-only base image containing a flat inode table and data area; an optional hash overlay that stores writable metadata and data through CAS-based structures; and an optional shared page cache whose slots and metadata also reside in the same DAX region. This supports several deployment modes.

| Mode | Layout | Role |
|---|---|---|
| Static | `[Super][Base]` | Read-only image |
| Empty | `[Super][Overlay][PCache]` | Writable overlay without base image |
| Split | `[Super][Base][Overlay][PCache]` | Read-only base plus writable overlay and shared cache |

The phrase “lock-free shared filesystem” has a specific meaning in DaxFS. In the multi-host data path, no kernel spinlocks, mutexes, distributed locks, or central lock managers protect data and metadata resident in CXL memory. Cross-host coordination for overlay updates, pool allocation, free-list management, page-cache state transitions, and eviction is performed solely through 64-bit `cmpxchg` on shared coherent cache lines, with memory barriers such as `smp_wmb()` and `smp_rmb()` to enforce publication and visibility ordering. Local per-host OS structures still use ordinary kernel locking, but those locks are not shared across hosts [2604.01620].

The access model is correspondingly direct. Reads and `mmap` resolve to CXL memory without per-host buffer caches, and for overlay-backed data the implementation uses per-inode xarrays local to each host to map offsets to DAX addresses. Read paths coalesce physically contiguous pages to reduce copy overhead, while `mmap` establishes PFN mappings directly to DAX pages so that userspace loads are issued against the shared memory itself [2604.01620].

## 3. CAS-based overlay, metadata model, and correctness discipline

The central writable structure in DaxFS is a CAS-based hash overlay layered over an optional immutable base image. All new or modified data and metadata are represented as overlay entries in a lock-free open-addressing hash table backed by a bump-allocated pool. Each bucket contains a 64-bit `state_key`, where bit 0 encodes FREE versus USED and bits 63:1 encode a 63-bit logical key, together with a `pool_off` field pointing to the actual object in the pool. Collisions are resolved by linear probing, and insertion proceeds by attempting a `cmpxchg` from `0` to `(key << 1) | 1` on the target bucket’s `state_key` [2604.01620].

The 63-bit key unifies several object classes. Data pages use
$$
\text{key} = (\text{ino} \ll 20) \mid \text{pgoff},
$$
supporting up to \(2^{20}\) pages, or 4 GB, per file. Inode entries use
$$
(\text{ino} \ll 20) \mid 0xFFFFF,
$$
directory-list heads use
$$
(\text{ino} \ll 20) \mid 0xFFFFE,
$$
and directory entries use a 63-bit `FNV-1a(parent_ino, name)` hash. The pool stores 32-byte inode entries, 4 KB data pages, directory entries of approximately 280 bytes, and 16-byte directory-list nodes; allocation is via a CAS-protected bump pointer, while each type also maintains a lock-free free list [2604.01620].

This organization gives DaxFS a simple update protocol. A writer claims a free bucket by CAS, allocates and initializes the corresponding pool object, executes `smp_wmb()` to guarantee content visibility, and only then publishes the `pool_off` pointer. Readers perform the inverse ordering: they read `pool_off`, execute `smp_rmb()`, and then access the pool object. The intended effect is that any reader observing a nonzero published offset also observes a fully initialized object. On x86 this primarily constrains compiler reordering under TSO; on ARM64 it materializes as architectural fences [2604.01620].

Directory and inode organization follow the same split design. The base image stores directories as flat arrays of fixed-size 271-byte directory entries containing name length, inode number, parent inode, and an inline name up to 254 bytes. The overlay maintains, for each directory, both a directory-list head entry and hashed individual directory entries; `readdir` enumerates base plus overlay, with overlay entries overriding base entries and tombstones representing deletions. File size is authoritative in the shared overlay inode entry, and each host refreshes its VFS inode size on the read path with `READ_ONCE()` rather than via explicit invalidation [2604.01620].

The paper’s correctness claim is not that conflicts disappear, but that conflicts are reduced to retries rather than inconsistency. If two hosts race to insert the same key, only one can win the FREE-to-USED CAS, and the loser observes a USED bucket with a matching key and treats the event as an update or duplicate insertion. Under well-provisioned load factors, linear probing eventually reaches either a matching key or a free slot. This is lock-free rather than wait-free: some participant always makes progress, but individual threads may retry [2604.01620].

## 4. Cooperative shared page cache and MH-clock eviction

When file data resides in a separate backing file, DaxFS uses a shared page cache, or `pcache`, located in the same DAX region and visible to all hosts. Each page-cache slot carries a 64-bit `state_tag` word and a `ref_bit`. The low two bits of `state_tag` encode FREE, PENDING, or VALID; bits [5:2] hold a 4-bit active-reader refcount from 0 to 15; and bits [63:6] hold a tag
$$
(\text{ino} \ll 20) \mid \text{pgoff}.
$$
By packing state, refcount, and tag into one 64-bit word, DaxFS makes slot state transitions CAS-addressable in a single atomic operation [2604.01620].

Cold-miss handling is coordinated without locks. A host probes up to eight candidate slots. If it finds a VALID slot with a matching tag, it increments the refcount by CAS and returns the slot. If it finds a FREE slot and no valid match, it attempts a CAS from FREE to PENDING for the desired tag. The winner reads the page from the backing file into slot memory with `kernel_read` and then CASes the slot to VALID. Other hosts that encounter a PENDING slot for the same tag spin until the state becomes VALID, thereby avoiding duplicate fills [2604.01620].

Eviction is handled by a novel multi-host CLOCK algorithm, MH-clock. In each host’s local probe window, victim selection first seeks slots with `state = VALID`, `ref_bit = 0`, and `refcount = 0`; if such a slot exists, the host CASes its `state_tag` to FREE. If all candidate slots are hot, indicated by `ref_bit = 1`, the host clears those bits with plain stores, yields briefly, and rescans. If no cold slot appears, it force-evicts the first VALID slot whose refcount is zero, irrespective of `ref_bit`. Because eviction is constrained by `refcount == 0`, slots currently in use by any host are not reclaimed [2604.01620].

MH-clock also includes a background sweep driven by a shared atomic `evict_hand`. A host may CAS-acquire ownership of a window of 64 slots and clear `ref_bit` values in that interval; hosts that lose the CAS skip the sweep. The effect is a distributed age-demotion mechanism that preserves the “second chance” behavior of classical CLOCK while avoiding centralized eviction control [2604.01620].

The same coordination model was extended experimentally to GPUs. DaxFS can expose its DAX region via `DAXFS_IOC_GET_DMABUF`, allowing the GPU driver to map the shared region as pinned host memory. GPU threads can perform pcache lookups by reading slot metadata and issuing `atomicCAS` for slow paths. On an RTX 5090 over PCIe 5.0, independent-slot 64-bit `atomicCAS` reached approximately 11.7 Mops/s at 512 threads, close to the reported PCIe 5.0 AtomicOp bandwidth limit of about 11.5 Mops/s. This does not constitute full GPU write support, which remains unimplemented, but it demonstrates that the atomics-only coordination model extends beyond CPUs [2604.01620].

## 5. Implementation, validation, and measured performance

The evaluation of DaxFS combines emulation-based correctness testing with DRAM-backed performance experiments. Multi-host correctness was validated in a patched QEMU 10.0 CXL 3.0 environment in which two virtual machines shared a GFAM-like region and CXL atomic operations were forwarded over TCP. This emulation is orders of magnitude slower than DRAM, but it exercises the CAS protocol under real concurrency. In those tests, DaxFS maintained more than 99% CAS “accuracy” under cross-host contention, and overlay insertions exhibited no lost updates even when both hosts created entries concurrently [2604.01620].

Single-host throughput experiments used DRAM-backed DAX memory as a stand-in for CXL memory. Platform A combined dual-socket Intel Xeon processors, 512 GB DDR5, and an NVIDIA RTX 5090 with a 512 MB DaxFS region allocated via CMA; Platform B used an Intel Xeon Gold 5418Y for comparisons against `ext4-dax` and `tmpfs`. The benchmark methodology used `fio` 3.x with synchronous I/O, 64 MB files, and filesystem reformatting between write experiments; the overlay was configured with a 400 MB pool and 65,536 buckets [2604.01620].

Several results are structurally important. For sequential writes, DaxFS exceeded `tmpfs` across all write workloads, including 1,730 MiB/s versus 1,362 MiB/s for a 4 KB first write and 2,783 MiB/s versus 2,560 MiB/s for a 1 MB rewrite. For random writes, the most prominent result was 4 KB random write throughput with four threads: 4,830 MiB/s for DaxFS versus 1,803 MiB/s for `tmpfs`, or 2.68× higher throughput. For random reads, DaxFS reached 15,052 MiB/s versus 12,800 MiB/s for `tmpfs` at 64 KB with four threads, a 1.18× improvement. Against `ext4-dax` on Platform B, DaxFS was 6.1× faster in 16 MB read latency, at 274.2 ms versus 1,660.7 ms, although `tmpfs` remained faster on some large sequential read cases [2604.01620].

| Workload | DaxFS result | Baseline comparison |
|---|---|---|
| Cross-host CAS under contention | `>99%` CAS accuracy | No lost updates in overlay insertions |
| 4 KB random write, 4 threads | 4,830 MiB/s | `tmpfs`: 1,803 MiB/s, `2.68×` lower |
| 64 KB random read, 4 threads | 15,052 MiB/s | `tmpfs`: 12,800 MiB/s |
| 16 MB read latency | 274.2 ms | `ext4-dax`: 1,660.7 ms |

The paper attributes these behaviors to the avoidance of per-page faults, folio accounting, page-cache locks, and inode xarray contention in paths where DaxFS instead uses pre-allocated DAX pages, lock-free overlay lookups, and direct pointer-based copies. The counterpoint is that DaxFS may pay per-page overlay resolution overhead on large sequential reads and may scan both base and overlay during `readdir`, where it was slower than both `tmpfs` and `ext4-dax` [2604.01620].

## 6. Limitations, related concerns, and significance

DaxFS makes several explicit trade-offs. Its directory representation is flat, with \(O(n)\) lookup behavior that is described as acceptable for modest sizes below 10K entries but expected to degrade for very large directories. The overlay hash table has a fixed size chosen at format time and does not support dynamic resizing. The pool does not implement compaction, tolerating fragmentation while relying on free-list reuse. Persistence assumes ADR or eADR and does not issue explicit `clflush` or `clwb`; no mechanism is provided for multi-host crash recovery, reconstruction after partial writes, or host failure in mid-operation. POSIX support is incomplete, with no `mknod`, FIFOs, sockets, or extended attributes, and filename length is capped at 255 characters [2604.01620].

A second misconception is that absence of journaling in the data path implies a general solution to durability and recovery. The paper states the opposite: the current prototype omits those mechanisms and treats crash consistency, persistence ordering on non-ADR systems, and full failure handling as future work. This suggests that DaxFS should presently be understood as a lock-free coordination and caching architecture rather than a complete answer to all persistence semantics in shared disaggregated memory.

There is also a broader systems concern inherited from DAX itself. Direct mappings make file access fast because subsequent operations are ordinary loads and stores, but that same property weakens syscall-centric observability. Prior work on hardware-assisted auditing for DAX-hosted filesystems argues that after the initial `mmap`, read and write activity within mapped regions becomes largely invisible to conventional OS-level auditing and provenance mechanisms [2104.08699]. DaxFS uses the same direct-access style for shared memory; this suggests that fine-grained auditability, provenance, and forensic logging remain open concerns for practical deployments, particularly in multi-host or accelerator-sharing environments.

Within the filesystem literature, DaxFS therefore represents a specific architectural thesis. Classical DAX reduces page-cache overhead but remains within a host-centric access model [1901.01340]. DaxFS goes further by assuming a coherent shared-memory fabric and collapsing multi-host filesystem coordination to CAS on that fabric. Its significance lies less in feature completeness than in the demonstration that coherent atomics can replace distributed locks, centralized coordinators, and per-host caches for an important class of shared-memory filesystems.

Source: https://www.emergentmind.com/topics/daxfs