---
title: 'ArceKV: Adaptive LSM Compaction'
url: https://www.emergentmind.com/topics/arcekv
type: topic
---

# ArceKV: Adaptive LSM Compaction

ArceKV is a workload-driven key-value store built atop RocksDB and introduced together with ElasticLSM and the Arce compaction decision engine to optimize Log-Structured Merge Tree performance under highly dynamic workloads [2508.03565]. It targets a setting in which recent LSM-tree research has focused on static workloads with fixed read-write ratios, while real-world workloads shift abruptly or smoothly over time. The system’s central claim is that traditional LSM-tree structural constraints impede continuous adaptation, and that removing those constraints expands the action space for compactions and write stalls, creating greater opportunities for sustained optimization during workload transitions.

## 1. Origins and problem setting

ArceKV is motivated by the observation that key-value stores underpin a wide range of applications due to their simplicity and efficiency, and that LSM-trees dominate as their underlying structure because they excel at handling rapidly growing data [2508.03565]. The paper positions the main systems problem not as static tuning, but as preserving high performance when workload patterns shift and when workload-aware methods must respond repeatedly rather than converge once.

The design addresses two limitations attributed to prior workload-aware approaches. First, approaches optimized for a fixed read-write mix often struggle to sustain optimal performance after workload changes. Second, methods that do adapt may incur substantial transition overhead when the workload pattern shifts. ArceKV is therefore framed as a system for continuous rather than episodic optimization: instead of waiting for a new steady state, it attempts to choose management actions that are immediately beneficial under the current workload trace.

This suggests a change in emphasis relative to conventional compaction design. Rather than treating the LSM structure as a target configuration to be restored or approximated, ArceKV treats compactions and write stalls as decision variables that should remain flexible as the state of the tree and the workload evolve.

## 2. ElasticLSM and the removal of structural constraints

The architectural basis of ArceKV is ElasticLSM, a design that removes traditional LSM-tree structural constraints to permit more flexible management actions, specifically compactions and write stalls [2508.03565]. Traditional LSM-trees are described as enforcing a fixed structural configuration that determines, for each level, the maximum number and size of sorted runs, the capacities of levels, and the conditions under which write stalls are triggered. ElasticLSM eliminates these restrictions.

The resulting policy is characterized in the paper as an **AnyTime–AnyRuns Policy**. Under this policy, compactions and write stalls can occur at any time and can involve any subset of sorted runs from across one or more levels, provided LSM-timestamp order is preserved. Levels may hold arbitrary numbers and sizes of runs, and the stall controller throttles writes only if the total number of runs exceeds a tunable threshold.

The practical significance of this relaxation is that the action space is no longer tied to a predefined structural template. In the paper’s example, when the workload shifts rapidly from update-heavy to read-heavy behavior, ElasticLSM allows a multi-level compaction of all runs to minimize lookups while also raising the write stall threshold. The intended effect is responsiveness without the pause or costly transition associated with converging toward a new rigid structure.

A concise comparison presented in the paper contrasts traditional LSM behavior with ArceKV’s policy model:

| System/Policy | Structural Constraints? | When/What to Compact |
|---|---|---|
| Traditional LSM | Yes | Predefined |
| ArceKV | No | Any runs, any time |

A related distinction concerns write stalling and transitions. Traditional systems use predefined thresholds and greedy, lazy, or fixed transition methods, whereas ArceKV uses a tunable threshold and continuous optimization [2508.03565].

## 3. Arce: adaptive compaction decision engine

ElasticLSM’s expanded action space requires a decision mechanism, which Arce supplies. Arce is described as a lightweight compaction decision engine that guides ElasticLSM in selecting the optimal action from its expanded action space [2508.03565]. Its purpose is to decide, in real time, which compaction and stall setting is most beneficial for the current tree state and workload.

The first stage is compaction candidate enumeration. The paper defines three valid compaction patterns:

- **Pattern 1**: Merge any subset of at least two runs within a single level.
- **Pattern 2**: Merge all runs from level $L$ and zero or more runs from $L+1$ into $L+1$.
- **Pattern 3**: Merge all runs across multiple consecutive levels, plus zero or more runs from the next level.

Candidate generation is then pruned heuristically by increasing run size and preferring larger reductions in run count for similar compaction cost. This pruning is part of keeping the search practical despite the larger action space.

The second stage is cost modeling. Arce divides operation sequences into windows, for example per MemTable flush, such that within a window the set of sorted runs, termed the tree state, is stable. Operation costs inside a window are defined by the following equations:

$$
\begin{align*}
P(s) &= (\alpha \cdot s + 1) \cdot I_r \\
R(s) &= s \cdot I_r \\
U(s) &= \frac{F}{B}\cdot I_w + k\cdot \mathbb{I}(s>c)
\end{align*}
$$

Here, $s$ is the number of sorted runs, $\alpha$ is the Bloom filter FPR, $I_r$ and $I_w$ are the I/O times to read or write a block, $F$ is the MemTable size, $B$ is the block size, $c$ is the write stall threshold, and $k$ is the stall rate per operation. The total cost over a simulation is given as

$$
C = \frac{\sum_{i=1}^{m} f(s_i, t_i)}{\sum_{i=1}^{m} t_i (r + u + p)}
$$

where $f(s_i, t_i)$ is the total cost in window $i$, $t_i$ is the duration in windows to complete compaction $i$, and $r$, $u$, and $p$ are the counts of range, update, and point operations per window [2508.03565].

The third stage is effectiveness scoring. Arce evaluates each action by combining a short-term penalty and a long-term benefit:

$$
E_s(s, t) = I_r \cdot t \cdot (r + \alpha \cdot p) + uk \cdot \max(0, s + t - c)
$$

$$
E_l(y) = (r + \alpha \cdot p) \cdot I_r \cdot y
$$

$$
E(s, t, y) = M \cdot E_l(y) - E_s(s, t)
$$

The short-term term captures the negative impact of long compactions, including stalls and delayed reads. The long-term term captures reduced lookup cost after the number of runs is decreased. Arce chooses the candidate compaction and stall configuration with the highest effectiveness score.

The decision algorithm is summarized as simulating multiple parameter triples $(M, c, k)$ and compaction sequences for the current workload and LSM state, calculating the average total cost for each, choosing the configuration with minimal average cost, and enacting its first recommended compaction or stall setting. The paper further states that the chosen compaction sequence achieves a 2-approximation of the global optimum average cost, and therefore strictly outperforms greedy or lazy-only strategies under the stated theorem [2508.03565].

## 4. Continuous adaptation under dynamic workloads

ArceKV’s main systems claim concerns dynamic workloads rather than static benchmark points. The paper emphasizes that existing systems such as greedy, lazy, or RL-based approaches either incur high stalls, respond slowly, or depend on accumulating sufficient feedback before adaptation becomes effective [2508.03565]. ArceKV instead attempts immediate, continuous, low-overhead optimization.

Three operational mechanisms support this claim. First, there is **no structural recomputation** in the sense used by systems that define a new target structure and then gradually shift toward it. Second, **workload-driven parameter update** is triggered only when the LSM state or workload changes by more than a threshold, with the default threshold given as $d=0.1$. Third, **responsiveness without stalls** is pursued by allowing aggressive merging in sudden read-heavy phases and by avoiding unnecessary compactions in write-heavy phases.

The paper’s treatment of transition overhead is central here. Existing approaches are described as suffering from either heavy stalls or slow response when structure changes are required. ArceKV is said to minimize transition overhead by always selecting actions that immediately optimize current workload cost, rather than requiring expensive global reorganization. The paper explicitly characterizes structural transitions as invisible to application workloads.

A plausible implication is that ArceKV redefines “adaptation” from a structural migration problem into a repeated online control problem over compaction scope and stall thresholds. That interpretation is consistent with the paper’s emphasis on the expanded action space and on immediate cost-aware selection.

## 5. Evaluation results and comparative behavior

The evaluation reports that ArceKV outperforms state-of-the-art compaction strategies across diverse workloads and delivers around $3\times$ faster performance in dynamic scenarios [2508.03565]. In compound workloads simulating abrupt or smooth read-write shifts, it consistently ranks first or near-first among six to seven baselines, including Leveling, Tiering, LazyLeveling, Moose, and Ruskey. The paper further states that it adapts rapidly, within 20 million operations, to workload shifts and can outperform RocksDB/1-Leveling, described there as the most adaptive baseline, by up to $3\times$ throughput.

Additional quantitative claims are reported for broader system comparisons. Under comparable conditions, ArceKV is stated to be more than $10\times$ faster than Cassandra and WiredTiger, and $3\times$ faster than CockroachDB and Pebble. In multi-threaded concurrent workloads, it scales better with increasing foreground or query threads than the baselines because it adjusts both compaction and write stalls responsively. On the YCSB suite, including point, range, read-update mixtures and workload skew, ArceKV achieves the best or nearly-best performance across the standard workloads; on YCSB-E, which is described as scan-heavy, it exceeds RocksDB throughput by more than $5\times$ [2508.03565].

The paper also discusses secondary metrics. Space amplification remains comparable to Leveling, although it is not explicitly optimized, which the authors attribute to effective merging and duplicate elimination. Simulation and parameter-search overhead average about $2\%$ of the total background task time. The simulation path is described as batched, parallelized, and pruned for efficiency, using SIMD and multi-core execution, with only a few milliseconds required per decision.

These results position ArceKV as a system whose gains are most pronounced when workloads change frequently enough that transition behavior, rather than steady-state asymptotics alone, determines end-to-end performance.

## 6. Relation to prior compaction strategies

The paper situates ArceKV against several families of compaction strategies rather than a single baseline [2508.03565]. Traditional LSM strategies are presented as structurally constrained. Greedy approaches are characterized as fast but prone to high stall or latency spikes. Lazy approaches are characterized as having low stall but poor responsiveness. RL-based approaches such as Ruskey are described as adapting through observed performance feedback, but their adaptation can be slow under read-heavy workloads because sufficient updates are needed before policy adjustment becomes effective.

ArceKV’s distinction is therefore not merely that it chooses different compactions, but that it removes the requirement to transition toward a predetermined configuration. Moose and Wacky are cited as examples of systems that require a new target structure and gradual movement toward it; by contrast, ElasticLSM plus Arce always acts in the immediate interest of the observed workload. This is the basis for the paper’s claim that ArceKV achieves both responsiveness and low transition cost.

The system can also be interpreted as separating **structural validity** from **performance optimality**. Traditional policies encode both into fixed rules, whereas ElasticLSM keeps only the validity condition that LSM-timestamp order be preserved and delegates optimization to Arce’s decision process. This suggests that the paper’s principal contribution is as much about search-space reformulation as about scoring.

## 7. Significance and limitations of scope

Within the scope defined by the paper, ArceKV is a principled attempt to make LSM compaction workload-driven at every step rather than only at convergence [2508.03565]. Its main contributions are the removal of fixed structural constraints through ElasticLSM, the formulation of a lightweight but theoretically justified decision engine through Arce, the use of simulation over windowed tree states, and the demonstration that these choices can improve throughput substantially under dynamic workloads.

The paper’s most explicit significance claim is operational: ArceKV continuously optimizes performance during workload transitions with minimal overhead and without latency spikes. For deployments in which read-heavy and write-heavy phases alternate, that objective differs materially from the objective of systems tuned for a single stable read-write ratio.

At the same time, the paper does not claim that every storage metric is explicitly optimized. Space amplification is reported as comparable to Leveling rather than as a primary optimization target. The emphasis remains squarely on compaction selection, stall control, and transition behavior. This suggests that ArceKV is best understood as a workload-driven control framework for LSM management whose novelty lies in expanded flexibility and near-optimal online decision-making, rather than as a wholesale redesign of key-value storage abstractions.

Source: https://www.emergentmind.com/topics/arcekv