Active Data Objects in MCAS

• Active Data Objects (ADOs) are user-supplied plugins that execute custom code in MCAS, enabling in-place manipulation of persistent memory values.
• ADOs integrate controlled in-store computation with pointer-based data structure updates, reducing client-server round trips and ensuring memory safety within a scoped execution domain.
• The approach supports continuous data protection workloads, yielding high throughput, low latency, and a reduced memory footprint compared to conventional standalone key-value designs.

Active Data Objects (ADOs) are a mechanism in MCAS (Memory Centric Active Storage) by which user-supplied plugin code is “pushed down” into the MCAS server and executed directly over persistent-memory value regions associated with one or more keys. In the reported design, MCAS is a persistent memory tier for high-performance durable data storage, implemented as a sharded, network-attached key-value store whose core index is a persistent hopscotch hash table. ADOs extend this store with in-place computation on persistent values, enabling arbitrary pointer-based data-structure manipulations such as trees, lists, and maps without moving raw data back to the client. The mechanism is presented together with a Continuous Data Protection (CDP) metadata workload that requires continuous updates to complex metadata under consistency and durability constraints (Waddington et al., 2021).

1. Conceptual role within MCAS

Within MCAS, an ADO is defined as a user-supplied plugin that executes custom code inside the MCAS back end, directly on the server’s persistent-memory value regions. The role of the mechanism is to associate in-store compute with a pool: clients invoke an ADO on a given key or key-range, and the plugin runs in the MCAS server, safely modifying the persistent memory of that pool. This organization is specifically intended to reduce data movement while preserving low-latency guarantees and data durability through memory persistence and replication (Waddington et al., 2021).

A central property of the mechanism is that it operates on value-memory “in place.” The persistent-memory regions are not merely passive storage for serialized values; they can hold complex pointer-based dynamic data structures under plugin control. This design supports use cases in which application logic would otherwise require repeated client-side fetch, decode, mutate, and write-back cycles. The paper’s CDP example shows that the intended target is not only simple key-value mutation, but also the maintenance of durable, application-specific state whose internal structure is managed by the plugin.

A common misconception is to treat ADOs as equivalent to unrestricted server-side execution. The reported system does not make that claim. Instead, the execution domain is scoped to a pool, and the ADO process only sees that pool’s persistent-memory mappings. This suggests that the mechanism is closer to bounded near-data execution over persistent state than to unconstrained code execution over the entire server address space.

2. Architectural organization

The MCAS substrate is sharded. Pools correspond to persistent-memory regions, implemented using device-DAX or fs-DAX together with a heap allocator, while transient metadata uses a separate DRAM allocator. One shard corresponds to one network endpoint and one set of CPU cores, and exclusively owns a set of pools. The key-value index is a persistent hopscotch hash table in persistent memory (Waddington et al., 2021).

The ADO extension adds a separate ADO process per pool. The MCAS shard and the ADO process memory-map the same persistent-memory regions. The shard exposes the key-value index and a callback API, and communicates with the ADO process over an IPC channel. The ADO process loads a user plugin as a shared object and executes plugin.do_work(...), reading and writing the value-memory in place. The client-side interaction is expressed as an invoke_ado(pool, key, opaque_request) operation, which is received by the shard and dispatched to the ADO process.

Threading is also explicit in the design. Each shard pins a primary ADO thread, while a small pool of secondary ADO threads is used for background work such as summarization and epoch garbage collection. In the CDP implementation, these secondary threads are important because summarization and aging-out are intentionally decoupled from the foreground update path.

The architectural consequence is a division of responsibilities. The shard remains responsible for networking, key-value indexing, and lock management, whereas plugin logic executes in a separate process over shared persistent-memory mappings. A plausible implication is that this separation is intended to preserve the performance properties of the MCAS fast path while still permitting application-defined computation over persistent data structures.

3. Invocation model and programming interface

The lifecycle of an ADO operation begins with registration and deployment. The user compiles an ADO plugin adhering to the IADO_plugin interface and registers it with MCAS under a pool name. On MCAS startup or pool-open, the shard forks or launches an ADO helper process and memory-maps the pool’s persistent-memory regions into that process (Waddington et al., 2021).

The client-side API includes invoke_ado(...) and invoke_put_ado(...), the latter first writing or resizing a value and then invoking the plugin. The principal invocation interface is:

$$commit\_latency = persist\_latency + replication\_latency$$5

Execution in the ADO process occurs through the do_work entrypoint:

$$commit\_latency = persist\_latency + replication\_latency$$6

The values parameter provides zero-copy handles, namely pointers and lengths, to the current value or values for the key. The plugin may also call back into the shard for memory management and key-value operations. The reported callback API includes create_kv, open_kv, erase_kv, resize_value, allocate_memory, free_memory, get_ref_vector, iterate, find_key, get_pool_info, and unlock_key.

This interface is notable because it combines in-place access to persistent values with controlled access to store-level services. The plugin is not limited to overwriting a single flat value. It can allocate memory, create or erase keys, iterate, and manage references, allowing persistent-memory-resident composite structures to be updated under application logic. The paper states that the model is generic and can layer encryption, replication, versioning, and domain-specific code such as graph traversals or ML kernels without changing the core store.

4. Memory safety, locking, and durability semantics

The reported safety boundary is the pool. An ADO process only sees its pool’s persistent-memory mappings, and key-level locks are held by the shard. By default, invoke_ado is exclusive on the target key unless flags request shared access. At the plugin level, do_work typically executes while owning the value’s memory, so in-plugin data-structure code does not need additional locking (Waddington et al., 2021).

Concurrency control is therefore layered. Each pool is assigned to exactly one shard. Within the shard, hash buckets are guarded by fine-grained locks, allowing concurrent invoke_ado calls to different keys to proceed in parallel. This is distinct from coarse store-wide serialization and is consistent with the multi-shard scaling results reported later.

Crash consistency is explicitly delegated to the plugin for mutating sequences. The stated requirement is that ADO plugins must ensure that any mutating sequence is either fully applied or fully rolled back on reboot. The typical approach is undo-logging in persistent memory:

1. write old-value snapshot to undo-log in PMem
2. flush(log_entry) (e.g., CLWB + SFENCE)
3. apply new value in place
4. flush(new_value)
5. zero (or trim) log entry

The durability model also includes synchronous replication. MCAS supports synchronous client-side replication in which the client will not send the next update until all $R$ replicas have acknowledged the prior update. The reported atomicity invariant is

$$commit\_latency = persist\_latency + replication\_latency$$

with the requirement

$commit\_latency \leq L_{max}$

where persist_latency is the time to CLWB+SFENCE the mutation to NVDIMM, replication_latency is the time for synchronous round-trip to $R-1$ replicas, and $L_{max}$ is the application’s latency SLO.

Failure recovery follows from these mechanisms. On restart, each shard scans the pools’ undo-logs, rolls back or commits partial updates, and resumes normal service. A common misunderstanding is that server-side execution alone guarantees crash safety. The design instead makes crash safety contingent on correct plugin-level persistence protocol, with the shard and recovery path enforcing the corresponding rollback or completion behavior.

5. Continuous Data Protection metadata as a worked example

The paper’s principal application is Continuous Data Protection metadata management. CDP records every block-range write over time, and the implementation uses a volume-level ADO whose persistent record type is

$$commit\_latency = persist\_latency + replication\_latency$$7

The design uses a two-level index. At the top level, an MCAS pool maps volume_tag to a 64-bit root pointer to volume ADO state. The per-volume state is stored entirely in persistent memory under plugin control (Waddington et al., 2021).

Updates are organized into a time-quantum list. Persistent_managed_range nodes are appended into a linked list of “quantum” chunks, where each chunk is a contiguous array of records with configurable size, for example 64 K entries = 4 MiB. When the active quantum becomes full, a secondary ADO thread performs lazy summarization: it reads the previous quantum’s summary in DRAM, replays the new quantum’s records in timestamp order to produce a new “point-in-time” summary map, such as a red-black tree or array, and atomically publishes the new summary back into persistent memory by swapping the root pointer and persisting it.

Point-in-time query processing at time $t$ is also explicitly defined. The procedure is: locate quantum $Q_t$ such that time(Q_t.timestamp) ≤ t; if $Q_t$ has no summary, scan backward for $Q_s$ with a summary; copy the summary of $Q_s$ into a result buffer $$commit\_latency = persist\_latency + replication\_latency$$0; replay all records in quanta from Q_s.next through $$commit\_latency = persist\_latency + replication\_latency$$1 whose timestamp is at most $$commit\_latency = persist\_latency + replication\_latency$$2 into $$commit\_latency = persist\_latency + replication\_latency$$3; and return $$commit\_latency = persist\_latency + replication\_latency$$4 to the client. Aging-out is handled by a background thread that scans the head of the quantum list and, if data are older than the retention threshold, unlinks the tail quantum and frees its memory.

This example clarifies what ADOs are intended to support. The update path, summarization path, query path, and aging path all manipulate persistent-memory-resident state whose structure is richer than a scalar key-value record. The CDP workload therefore demonstrates ADOs as a substrate for durable, mutable metadata structures with both foreground and background computation.

6. Measured behavior and comparative results

For the CDP workload with a 100% write workload and quantums of 64 K recs, the reported write throughput on a single shard with a single client thread is 140 K updates/sec with mean RTT ≃ 7.14 μs. Scaling the number of shards up to 12, using one client and 6 threads per client, reaches 4.92 M updates/sec. Under replication, the same workload yields 3.92 M/sec for two-way replication, described as ≈ 20% ↓, and 3.07 M/sec for three-way replication, described as ≈ 37% ↓ (Waddington et al., 2021).

Latency distributions on a single shard remain concentrated below 10 μs: with 1 thread, 99.77% of operations < 10 μs; with 6 threads, 99.39% < 10 μs. For point-in-time query latency with 100 K blocks output, quantum sizes of 4 MiB (65 K recs), 8 MiB, and 16 MiB are evaluated, and the worst-case latency rises by ≈ 50 ms per additional 4 MiB of quantum. When periodic queries are issued from a second client, writer throughput drops from 144 K upd/sec to 127 K upd/sec, corresponding to 11.8% degradation.

The comparison with a “Plain KV” thick-client design is particularly informative. In that baseline, the client performs all CDP logic locally and MCAS is used only for storing atomic update frames. The ADO solution shows higher throughput by 43% at 2 threads/client and by 30% at 1 thread/client. Mean write latency is ≃ 6.7 μs for ADO versus ≃ 16.6 μs for Plain-KV, and 99.97% of ADO writes are below 20 μs versus 94.27% for Plain-KV.

For memory footprint on a single shard with 16 MiB quantum and keep 10 quanta, the reported values are as follows:

Component	ADO	Plain-KV
Client DRAM	912 MiB	1747 MiB
Server DRAM	530 MiB	651 MiB
Server PMem	327 MiB	655 MiB
Total Footprint	1.72 GiB	2.98 GiB

The corresponding increases for Plain-KV relative to ADO are ↑ 91% for client DRAM, ↑ 23% for server DRAM, ↑100% for server PMem, and ↑72% for total footprint. These results support the paper’s claim that ADOs reduce round-trip data movement for complex pointer-based updates by executing near the persistent data, and they suggest that the performance benefit is coupled not only to lower latency but also to reduced memory footprint in the evaluated workload.