OpenCXD: Hybrid Evaluation for CXL-SSDs

• OpenCXD is a real-device-guided hybrid evaluation framework that integrates cycle-accurate host simulation with hardware firmware execution to capture realistic CXL-SSD behaviors.
• It overcomes the limitations of software-only SSD simulators by exposing firmware-level and NAND variability through empirical latency measurements.
• The framework enables performance optimization and design insights into firmware scheduling, buffering, and parallelism in memory-semantic CXL-SSD systems.

OpenCXD is a real-device-guided hybrid evaluation framework for CXL-SSDs that couples a cycle-accurate CXL.mem-enabled host simulator with a physical OpenSSD platform running real firmware. It was introduced to address a specific methodological gap in CXL-SSD research: host-side CXL.mem latency can be modeled reasonably in simulation, but firmware-level interactions, NAND controller behavior, and low-level storage dynamics are not faithfully captured by software-only SSD simulators. OpenCXD therefore retains full-system host simulation while replacing the SSD simulator with a programmable hardware device that executes actual firmware and returns per-request latency measurements to the simulator timeline (Chung et al., 15 Aug 2025).

1. CXL-SSD context and the problem OpenCXD addresses

Compute Express Link is a cache-coherent, high-bandwidth interconnect layered on PCIe. Among its three sub-protocols, CXL.mem exposes device memory into the host’s physical address space and allows CPUs to access it using ordinary load/store operations. In a CXL-SSD, SSD capacity is presented via CXL.mem as a memory-semantic device rather than a block device: the host issues 64-byte cacheline accesses, the controller buffers them in on-board DRAM, and the firmware aggregates them into NAND-page operations such as 16 KiB page writes (Chung et al., 15 Aug 2025).

This memory-semantic presentation introduces a structural mismatch between host cacheline granularity and NAND flash page granularity. Prior CXL-SSD designs such as SkyByte address this with firmware structures including a Write Log, a Data Cache, a two-level Log Index, and periodic log compaction. These mechanisms make flash-based storage behave more like a large, slower memory tier, but they also move firmware and controller behavior directly into the critical path of memory access latency.

Before OpenCXD, most CXL-SSD evaluation relied on software-only hybrid simulators. On the host side, tools such as MacSim or gem5 were extended to inject CXL.mem latency and bandwidth for a designated memory region. On the device side, SSD simulators such as SimpleSSD or FlashSim approximated NAND timing, FTL behavior, and controller scheduling. This approach is useful for early exploration, but it has well-defined blind spots. The paper highlights static latency models for NAND and firmware, lack of real firmware execution, and the inability to reproduce per-request variability, latency spikes, and dynamic interactions among DRAM, NAND, and ARM cores inside the device. OpenCXD is explicitly designed to replace those simulated SSD-side assumptions with in-situ firmware execution on hardware.

2. Architecture and component organization

OpenCXD has a two-sided architecture: a simulated host and a real device. The host side is a MacSim-based x86 system simulator extended with CXL.mem semantics and a CXL-SSD-aware memory subsystem. The device side is an OpenSSD DaisyPlus platform that runs real SSD firmware on an ARM-based SoC (Chung et al., 15 Aug 2025).

Side	Component	Role
Host	MacSim-based x86 simulator	Models pipeline, caches, DRAM, and CXL.mem access redirection
Device	DaisyPlus OpenSSD	Executes real firmware and measures device latency
Interface	Custom NVMe command path	Carries simulated CXL.mem requests to hardware

The DaisyPlus board used in the framework includes a Xilinx Zynq UltraScale+ SoC with ARM Cortex-A53 cores, 2 GiB LPDDR4 DRAM, 256 GiB NAND organized as 4 channels × 8 ways with 16 KiB pages, and a PCIe Gen3 ×16 interface. The firmware stack contains classical SSD components—Host Interface Layer, Flash Translation Layer, and Flash Interface Layer—along with CXL-SSD-specific modules such as the Write Log, Data Cache, Log Index, and log compaction.

This organization is central to OpenCXD’s purpose. The host remains fully observable and configurable at cycle level, while the device executes firmware under real timing conditions. The framework is therefore neither a pure simulator nor a pure hardware testbed. It is a hybrid system in which the host-side architectural model and the device-side firmware/hardware path are bound together at the execution boundary of each CXL-SSD access.

3. Request path, timing model, and execution semantics

When the simulated CPU issues a load or store to an address mapped to the CXL-SSD region, OpenCXD translates that memory event into a real device interaction. The workflow has seven stages (Chung et al., 15 Aug 2025):

1. LLC miss in the x86 simulator: MacSim models the CPU pipeline and caches until a memory access misses in the LLC.
2. Address range check: the simulator determines whether the physical address belongs to ordinary DRAM or to the CXL-SSD region.
3. Encoding into a custom NVMe command: because DaisyPlus speaks NVMe rather than native CXL.mem, OpenCXD defines custom NVMe commands that carry the target memory address, the operation type, and opcode bits indicating the CXL-SSD path.
4. NVMe passthrough: the host OS NVMe driver sends the command using libnvme/ioctl, and the x86 simulator pauses its cycle counter.
5. On-device firmware execution: the SSD firmware executes the corresponding CXL-SSD logic. For writes, this includes Write Log insertion, Data Cache update when relevant, and Log Index maintenance. For reads, the firmware checks the Data Cache, then the Write Log, and finally NAND through FIL if necessary. Periodic log compaction merges buffered writes back into NAND pages.
6. Latency return via CQE: the firmware embeds the measured start-to-end latency into a reserved field of the NVMe Completion Queue Entry.
7. Simulator resumption: the host extracts the latency, adds configured CXL.mem interface overhead, converts the result into cycles, advances the simulation clock, and resumes instruction execution.

The conceptual access-latency decomposition is

$$T_{\text{access}} = T_{\text{host}} + T_{\text{CXL.mem}} + T_{\text{device}}$$

where $T_{\text{host}}$ is the host-side execution and cache-hierarchy latency up to the LLC miss, $T_{\text{CXL.mem}}$ is a fixed or configurable interface delay such as 40 ns, and $T_{\text{device}}$ is the end-to-end SSD-side latency measured in hardware. The conversion back into simulation time is

$$\text{cycles\_added} = \frac{T_{\text{device}} + T_{\text{CXL.mem}}}{T_{\text{cycle}}}$$

with $T_{\text{cycle}}$ set by the simulated CPU frequency.

A key methodological property follows from this design. Host-side timing remains cycle-accurate within MacSim, while each CXL-SSD access introduces an atomic delay derived from real firmware execution on the hardware device. This is the defining “device-in-the-loop” feature of OpenCXD.

4. Firmware and NAND behavior that simulation-only methods miss

The principal contribution of OpenCXD is empirical rather than merely architectural: it exposes device-side effects that software-only SSD simulators either suppress or approximate too aggressively. The paper emphasizes both NAND-level variability and firmware-level variance (Chung et al., 15 Aug 2025).

For NAND I/O, experiments using fio at page granularity on SK Hynix and Toshiba NAND show large latency variance on real hardware, especially at higher queue depth. For SK Hynix NAND, at iodepth = 1, the standard deviation of read time $t_R$ is approximately 1.1 µs and the standard deviation of program time $t_{Prog}$ is approximately 37.61 µs. At iodepth = 8, the standard deviations rise to approximately 974 µs for $t_R$ and approximately 1111 µs for $t_{Prog}$. By contrast, SimpleSSD exhibits essentially zero variance for many parameters and only approximately 11 µs deviation for $T_{\text{host}}$0 under scheduling. The paper further reports that real device NAND read latencies are on average 2.4× higher than simulation-based values once controller and firmware overheads are included. Toshiba NAND also shows brief spikes up to approximately 440 µs, illustrating device-specific signatures absent from simplified models.

Firmware-side operations also deviate from static assumptions. In SkyByte’s simulation, write log insert latency is fixed at 640 ns and DRAM cache hit latency at 712 ns. OpenCXD instead observes workload-dependent latency variation and reports that write log insert and DRAM cache hit latencies often exceed the 2 µs context-switch threshold used by SkyByte. At finer granularity, for the srad and ycsb workloads, average DRAM cache operations are approximately 32–37 ns with standard deviation around 30 ns, while Write Log index checks average approximately 171–183 ns with standard deviation 30–55 ns. The paper interprets this as evidence that even ostensibly “cheap” DRAM-side firmware operations have nontrivial variance and that heavier index-related logic is materially more expensive than static models imply.

The same divergence appears in latency distributions for NAND-backed reads. In SkyByte’s SimpleSSD-based model, a single latency bucket at approximately 99.72 µs accounts for 87.2% of reads in srad and 94.3% in ycsb. OpenCXD instead shows broad, workload-specific distributions with significant tails, reflecting controller scheduling, queueing delays, and internal parallelism. Because CXL-SSDs expose memory-semantic cacheline accesses to the CPU, these distributional differences translate directly into different observed memory-stall behavior at the processor.

5. Experimental methodology, results, and design implications

The evaluation uses a MacSim host configured with 8 Skylake cores and up to 3 threads per core, replaying SkyByte memory traces for seven workloads: bc, bfs-dense, dlrm, radix, srad, tpcc, and ycsb. Each workload includes up to 1 million memory accesses, except bfs-dense, for which the full trace is used. DRAM and NAND parameters in SkyByte’s simulator are tuned to match DaisyPlus hardware, and both platforms perform pre-filling and warm-up (Chung et al., 15 Aug 2025).

Metric or case	Simulation-only view	OpenCXD observation
Write log insert / DRAM cache hit	Static 640 ns / 712 ns	Workload-dependent, some operations exceed 2 µs
SSD DRAM miss / NAND read	Narrow and mostly fixed	2.4× higher average latency across workloads
Log compaction	Sequential baseline	Up to 8× faster with NAND-parallel compaction

Several results are central.

First, OpenCXD reports higher CPU cycles per instruction than SkyByte for all evaluated workloads. The reason given is that CXL-SSD reads are slower and more variable under real-device execution, while context switching over 3 threads per core cannot fully hide NAND delays. This directly challenges optimistic host-side conclusions drawn from static SSD timing models.

Second, OpenCXD supports firmware experimentation on real hardware. The paper’s case study restructures log compaction to exploit channel-level NAND parallelism. Instead of compacting each page sequentially, the optimized method pre-scans indices, batches pages, and issues parallel reads and writes across channels. The reported result is up to 8× speedup in log compaction time across various Write Log sizes.

Third, the authors extract broader design implications. Firmware scheduling and buffering must account for realistic NAND latency variance rather than nominal page timings; DRAM-side operations such as Write Log, Data Cache, and Log Index processing are not negligible; host-side context-switch thresholds should be co-designed with measured device behavior; and NAND parallelism is powerful but requires firmware structures that batch work in alignment with channel and die topology. A plausible implication is that CXL-SSD architecture evaluation based solely on static SSD-side models can materially misestimate both processor-visible latency and the efficacy of host-side policies such as context switching.

6. Openness, limitations, and place in CXL-SSD research

OpenCXD is explicitly described as open. Its source code is publicly available at https://github.com/hschung1652/opencxd, it is built on OpenSSD platforms such as Cosmos+ and DaisyPlus, and the MacSim-based host simulator is also open-source (Chung et al., 15 Aug 2025). This matters because the framework is intended not merely as a one-off evaluation environment but as a reusable platform for firmware and architecture research. Researchers can replay memory traces, modify firmware modules, vary Data Cache or Write Log policies, experiment with alternative Log Index structures, and adjust CXL.mem latency and bandwidth parameters.

The framework nevertheless has clearly stated limitations. It does not use a native CXL.mem device; instead, it emulates CXL.mem semantics over NVMe custom commands. Actual device-level CXL controller behavior, including link retry, ordering rules, and coherence interactions, is therefore outside the hardware path and is represented host-side as a fixed parameter. The current implementation also has limited multi-request concurrency on the SSD side because NVMe passthrough is effectively sequential, so highly parallel designs may be underestimated. Coverage is limited to CXL.mem rather than CXL.cache, CXL.io, or multi-device topologies, and the device-specific behavior observed on DaisyPlus with its NAND configuration may not generalize without porting to other controllers and flash technologies.

Within CXL-SSD research, OpenCXD occupies a distinctive position. Earlier work established the architectural attractiveness of presenting flash-backed capacity as memory-semantic storage, but those studies necessarily relied on software approximations for the device side. OpenCXD shifts the evaluative center of gravity from parameterized SSD models to measured firmware execution on real NAND hardware. Its significance is therefore methodological as much as architectural: it provides a bridge between host-side cycle simulation and device-side empirical behavior, enabling co-design of CXL-SSD firmware, controller policies, and host memory-management strategies under conditions that are materially closer to deployable systems than simulation-only setups.