ForesightKV: Predictive-Scheduling Key-Value Database

• ForesightKV is a predictive-scheduling deterministic key–value system that combines statistical conflict prediction with multi-version storage for serializable transactional workloads.
• It employs an ASPN model to accurately predict conflicts and an MTFS algorithm to efficiently reorder transactions, ensuring high commit rates under contention.
• The system demonstrates robust performance improvements, achieving up to 86% commit rates and significant throughput gains on benchmarks like TPC-C and YCSB-A.

ForesightKV is a predictive-scheduling deterministic key–value database system that integrates statistical conflict prediction with multi-versioned storage and advanced scheduling for serializable transactional workloads under high contention. It is distinct from heuristic-based approaches by using Association Sum-Product Networks (ASPN) to predict conflicts, Matrix Two-Pass Forward Scan (MTFS) to reorder transactions, and multi-version concurrency control (MVCC) for high commit rates, while providing deterministic, replication-friendly execution. The system is designed to maximize throughput and scalability on transactional and key–value workloads, particularly with non-declared access patterns and skewed contention profiles (Huang et al., 24 Aug 2025).

1. System Architecture and Workflow

ForesightKV is organized into a five-layer deterministic pipeline: Input, Sequencing, Scheduling, Execution, and Storage. The Input layer accepts SQL or key–value transactions from clients without requiring prior extraction of read/write sets. The Sequencing layer employs an ASPN-based conflict predictor and the MTFS reordering algorithm to analyze transaction dependencies and produce both a conflict-minimized deterministic execution order and an abort set for transactions with strong conflicts.

The Scheduling layer validates this batch order against Rule 1 (no read-after-write [RAW] dependencies) on a consistent snapshot, committing all non-conflicting transactions and deferring others to the Fallback phase. The Execution Engine enacts speculative execution for each transaction on the batch snapshot, buffering writes as private version chains. The Storage Engine utilizes MVCC, tracking all version chains for each key and garbage-collecting obsolete versions using epoch-based reclamation.

The overall workflow proceeds as follows: after batch submission and conflict prediction, the MTFS scheduler reorders transactions and identifies an abort set. Non-aborted transactions are executed in parallel and committed by appending atomic version chains. Deferred transactions are processed in timestamp order, always reading visible versions per their snapshot isolation, and committed without locks.

2. Association Sum-Product Network (ASPN) for Conflict Prediction

ForesightKV’s ASPN model addresses the challenge of predicting transactional conflicts without prior access to read or write sets. Given two transaction predicates (Q₁, Q₂), ASPN infers the probability that transactions will access overlapping attributes, i.e., the likelihood of conflict. Each node in the ASPN is defined as $N_n=(A_n, C_n, T_n)$ for a target attribute set $A_n$, conditional context $C_n$, and data $T_n$.

Node types are:

• Decomposition ($\oplus$): splits data into k clusters with mixture weights.
• Independent ($\times$): factors probability as a product of marginals if attributes in $A_n$ are independent.
• Joint ($\boxempty\boxempty$): captures strong correlation with direct joint estimation.
• Leaf ($\boxempty$): fits univariate marginals for single attributes.

ASPNs are recursively constructed using RDC-based dependency search and clustering, yielding a tree whose leaves represent attribute marginals or joint distributions. Training cost is $O(n|A|^2 r \log r + d b + |T|)$, where n is the number of nodes, d is the number of decompositions, b is branch factor, and r is sample size.

For inference, ASPN traverses the tree with the intersection query $Q^*$, aggregating probabilities. Conflict likelihood simplifies to the probability of write/read overlap: $$p^{\rm RAW}_{ij} = \Pr_T(W(T_i)\cap R(T_j)) \approx \Pr_T(Q_i^{\sf write}\cap Q_j^{\sf read})$$ ASPN’s prediction accuracy is reported at 95–99% (precision 93–100%) with sub-2 second inference, incurring less than 3% added overhead per batch (Huang et al., 24 Aug 2025).

3. Multi-Version Storage and Validation

ForesightKV employs a multi-version storage engine, supporting MVCC to resolve write-after-write (WAW) and write-after-read (WAR) conflicts, while relaxing the validation requirement to focus solely on RAW dependencies. Each record maintains a chain of versioned values, indexed by transaction ID. During execution, reads select the appropriate version at or before their snapshot, and writes generate private version nodes.

Transactions are only aborted or deferred if a RAW dependency exists with a lower-TID transaction (Rule 1). Aborted transactions are re-executed in a fallback phase, accessing merged version chains and respecting strictly increasing commit order.

Epoch-based garbage collection ensures that obsolete versions (including those from aborted transactions) are safely reclaimed once the minimum active TID has advanced. This architecture yields significant improvement in commit rates and throughput. In TPC-C experiments, baseline single-version scheduling has a commit rate ≈ 0.69, predictive scheduling raises it to ≈ 0.98, and the full ForesightKV stack achieves up to 86% commit rate versus 19% for Aria under high contention (Huang et al., 24 Aug 2025).

4. MTFS: Matrix Two-Pass Forward Scan Scheduling

The MTFS algorithm enables global schedule reordering by using a compact dependency matrix and a two-pass forward scan for extracting the minimum necessary abort set—a feedback vertex set (FVS)—to maximize concurrent commits.

First, the dependency matrix $M\in\mathbb{N}^{N\times N}$ is constructed, encoding all direct and indirect dependencies: $$M[i, i] = 1,\quad M[i, j]=1 \text{ for all direct predecessors } j \in D[i]$$ Rows are incrementally updated to accumulate all paths between transactions.

Second, MTFS uses RAW edges to enumerate paths that would produce cycles, counting the frequency of each transaction appearing in those paths. The highest-frequency nodes are selected into the abort set $S$, breaking cycles and yielding a topologically sortable, acyclic schedule for maximized safe commit.

For batch sizes $N=500\!-\!1000$, MTFS scheduling runs in sub-millisecond times (complexity $O(N^2)$). In TPC-C, ForeSightKV’s single-threaded MTFS achieves 8.2k txn/s at 0.861 commit rate, and parallel MTFS achieves 17.1k txn/s at 0.847, outperforming alternatives such as Aria and FGA (Huang et al., 24 Aug 2025).

5. Performance and Skew Robustness

ForesightKV demonstrates strong throughput and scalability advantages, particularly on skewed, high-contention workloads. On YCSB-A, it delivers 260k txn/s (uniform), and retains 76% of this baseline under extreme Zipfian skew ($\theta=0.999$), where Aria drops to 34% and AriaFB to 58%. For partitioned TPC-C, ForesightKV achieves ≈120k txn/s for 20+ partitions, compared to Aria’s 80k.

Reordering via MTFS provides substantial gains: in TPC-C with 20 warehouses, ForesightKV commits ≈86% of transactions, versus 19% for Aria, and shows ≈4.6× throughput under heavy skew ($\theta \to 0.9$). ASPN prediction incurs minimal overhead (<3%). The system is deterministic, suitable for geo-replication, and does not require heuristics or pre-extracted access patterns (Huang et al., 24 Aug 2025).

6. Applicability, Limitations, and Tuning

ForesightKV is particularly well-suited for geo-distributed deployments, real-time analytics, and high-contention key–value and transactional workloads. Its strengths include independence from a priori access set extraction, high commit rates in skewed/contended scenarios, and batch-wise deterministic execution.

Limitations center on ASPN construction and retraining costs (dependent on sample size) and the $O(N^2)$ space/time requirements of MTFS for very large batches. There is a modest increase in end-to-end transaction latency due to batch sequencing compared to fully two-phase locking (2PL) based protocols.

Recommended tuning parameters include: conflict threshold $\tau$ in [0.1, 0.2] for abort decisions; ASPN sample size $r=10\,\text{k}\,\!-\!50\,\text{k}$ records per table for >90% precision under 60s of training; batch size $N=500\!-\!1000$ for throughput/overhead balance; and garbage collection epochs set to double the maximum batch latency to prevent version retention bloat.

ForesightKV’s design pattern of combining probabilistic conflict modeling, multi-versioned snapshotting, and batch global reordering concretely achieves up to 2× throughput gains under skew, >80% commit rates even with contention, and deterministic, serializable schedules without the shortcomings of double execution or coarse-grained locking (Huang et al., 24 Aug 2025).