MarlinCommit: Cloud DBMS Coordination

• MarlinCommit is an optimized atomic commit protocol designed for storage-disaggregated, partitioned-writer cloud DBMSs, committing only if no cross-node modification has occurred since the last observed commit.
• It leverages a CAS-like conditional append mechanism and log participation to safely handle concurrent cross-node updates, ensuring strong ACID properties and serializability for reconfiguration transactions.
• The protocol enhances performance and fault tolerance by eliminating external coordination services, as evidenced by superior migration throughput, reduced latency, and efficient failover management compared to ZooKeeper-based solutions.

MarlinCommit is the commit protocol introduced in "Marlin: Efficient Coordination for Autoscaling Cloud DBMS (Extended Version)" (Hu et al., 3 Aug 2025). It is designed for a storage-disaggregated, partitioned-writer cloud DBMS in which some coordination metadata may be modified by multiple compute nodes, and it is defined as “an optimized atomic commit protocol that commits a transaction only if no cross-node modification has occurred since each node’s last observed commit.” In Marlin, this protocol is the mechanism that allows the system to eliminate an external coordination service such as ZooKeeper while preserving strong transactional guarantees for failover-critical metadata and ordinary application state.

1. Architectural setting

Marlin is motivated by a shift in modern cloud databases from converged architectures to storage disaggregation, which enables independent scaling and billing of compute and storage. The paper argues that cloud databases still commonly rely on external, converged coordination services for the control plane, and that these services inherit limitations analogous to those that motivated storage disaggregation in the data plane: scalability bottlenecks, low cost efficiency, and increased operational burden. Marlin therefore disaggregates cluster coordination and consolidates coordination functionality into the same cloud-native database it manages (Hu et al., 3 Aug 2025).

The system context is a cloud-native OLTP DBMS with storage disaggregation, partitioned-writer architecture, and log-as-the-database storage. The storage layer provides durable WALs, asynchronous page materialization, Append(updates), GetPage(pageId, [LSN](https://www.emergentmind.com/topics/visual-ladder-side-network-lsn)), and Marlin’s enhanced conditional append Append(updates, LSN). The compute layer consists of stateless compute nodes that process transactions, cache pages, and track the highest committed LSN they know for each relevant log.

Within this architecture, Marlin distinguishes application state from coordination state. Application state is ordinary user data, partitioned across compute nodes and normally accessed exclusively by the owner node. Coordination state is metadata needed to manage the cluster. Two structures are central. MTable stores group membership, including node IDs and addresses, in a single global log MLog. GTable stores granule ownership mappings from granule to owner node; it is partitioned by owner node and stored in per-node GLogs. Other metadata unrelated to failover is managed like ordinary partitioned user data.

The architectural novelty creates the condition that motivates MarlinCommit. In a conventional partitioned-writer design, concurrency control and commit protocols assume that a node has exclusive control over the data it manages. In Marlin, some critical coordination states, specifically membership metadata and failover-critical ownership metadata, can be accessed and modified by multiple nodes. This shared writability is necessary during reconfiguration and failover, but it invalidates the exclusivity assumption underlying ordinary commit processing.

2. Cross-node modification as the central problem

The paper uses cross-node modifications to denote updates to the same coordination state or log by different compute nodes. This is not a generic statement about distributed transactions over multiple partitions. The issue is specifically that certain coordination structures remain writable even when the nominal owner compute node is unavailable, so that failover can proceed without waiting for that node to recover (Hu et al., 3 Aug 2025).

Several examples illustrate the problem. Multiple nodes may concurrently update MTable, as in AddNodeTxn and DeleteNodeTxn. If a node fails or is partitioned, another node may update that failed node’s GTable partition during RecoveryMigrTxn. If the supposedly failed node later returns with stale metadata, then both nodes may attempt operations against the same ownership metadata. In such cases, local concurrency control on a single node is insufficient because the relevant race spans multiple compute nodes and the same metadata log.

This is also why removing an external coordination service creates new commit challenges. With an external coordination service, consistency of metadata is handled by that service, typically through a leader and consensus or replication, and the database nodes consult it rather than sharing mutable control metadata directly. In Marlin, compute nodes are stateless and not replicated, durability and replication reside in the storage layer, and failover-critical coordination state resides in disaggregated storage. Another node may therefore need to commit updates affecting metadata that would ordinarily be regarded as belonging to a different node’s log. MarlinCommit is the mechanism that makes such updates safe.

The design goals stated for MarlinCommit are correspondingly specific: strong transactional guarantees, data consistency under concurrent reconfiguration, user transactions, and failures, safe failover without an external coordination service, support for cross-node modifications on coordination state, efficiency in storage-disaggregated environments, compatibility with existing transaction managers and protocols, scalability for partitioned metadata such as GTable, and fault tolerance including commit progress even if a compute node is unresponsive.

3. Core mechanism: conditional append and log participation

The core storage primitive in MarlinCommit is the conditional append API, also called Append@LSN:

$$(status, new\_lsn) \leftarrow RPC_{sync/async}^{log::Append(updates, target\_lsn)}$$

This operation appends only if the target log is still at the expected version target_lsn; otherwise it fails and returns the current log position. Each compute node maintains local knowledge of committed log positions. The paper describes a local H-LSN as the highest LSN successfully appended and, more specifically for MarlinCommit, an lsn_tracker array that records the last committed LSN of each log in the cluster. If the tracked LSN matches the log tail, the append succeeds and yields a new LSN. If not, some other append has occurred, and the operation fails. In MarlinCommit, failure of this conditional append is interpreted as evidence that “a cross-node modification on the same log may have occurred” (Hu et al., 3 Aug 2025).

This mechanism gives the protocol a compare-and-swap style basis. The paper states that the protocol is storage-agnostic as long as the storage service supports CAS-like conditional writes, for example through ETags and If-Match, object generation numbers, or append-position conditions. It explicitly notes that this can be implemented on Azure Blob Storage, Amazon S3 variants, Google Cloud Storage, Bigtable-like systems, and DynamoDB-like systems, provided that a suitable CAS exists.

A second distinctive feature is the participant model. In MarlinCommit, “a participant can be either a compute node or a log instance in the disaggregated storage.” MLog and node.GLog are direct examples. The paper gives two reasons for this design: “the log is the ground truth of the system” and “voting through a node is semantically identical to appending the vote directly to the log.” This departs from conventional 2PC, in which participants are generally process-level transaction managers rather than durable logs.

As a result, MarlinCommit is not simply ordinary 1PC or 2PC layered over logs. The paper characterizes it as generally following the conventional commit protocol but with two key differences: it replaces ordinary direct log append with TryLog, which performs conditional append and stale-metadata invalidation, and it allows logs themselves to participate in the commit protocol.

4. Commit procedure and transaction classes

MarlinCommit supports both one-phase and two-phase style execution depending on the participant set. For a single participant, it performs a one-phase commit through TryLog. For multiple participants, it follows a 2PC-like structure: the coordinator contacts participants asynchronously, log participants are handled by direct TryLog(log, {VOTE-YES ∪ updates[p]}), compute-node participants receive VOTE-REQ, the coordinator collects responses, computes a final decision, and broadcasts that decision asynchronously (Hu et al., 3 Aug 2025).

The helper procedure is central:

Function TryLog(log, updates) { (status, new_lsn) ← RPCsync^log::Append(updates, lsn_tracker[log]) if status is FAILURE { ClearMetaCache(log) lsn_tracker[log] ← new_lsn return ABORT } else { lsn_tracker[log] ← new_lsn return COMMIT } }

The high-level behavior proceeds in stages. The transaction first executes its normal logic, including state checks, lock acquisition when needed, and preparation of updates to application state and coordination state. The coordinator then builds the participant set. The examples given in the paper include AddNodeTxn on {MLog}, MigrationTxn on {src, dst}, RecoveryMigrTxn on {src.GLog, dst}, and ScanGTableTxn over MLog ∪ nodes. The paper notes that the notation is somewhat inconsistent across the text and algorithm, but the essential point is that the participants are the nodes or logs whose durable commit records are required.

TryLog performs the decisive operation. If Append(updates, lsn_tracker[log]) succeeds, the node updates its tracker and returns COMMIT. If it fails, the node clears the relevant metadata cache, updates the tracker to the returned new_lsn, and returns ABORT. ClearMetaCache(MLog) evicts the MTable cache; ClearMetaCache(node.GLog) evicts the corresponding GTable partition cache. This makes stale metadata visible as an abort condition rather than allowing it to persist silently across transactions.

After such an abort, the next transaction that misses in the system-table cache fetches fresh state from storage using GetPage@LSN, guided by the updated lsn_tracker. The protocol’s core rule is therefore direct: a transaction commits if and only if no relevant log has been modified by another node since the local last-known commit; otherwise it aborts.

The protocol’s role in specific Marlin transactions is especially visible in reconfiguration. AddNodeTxn and DeleteNodeTxn update MTable through globally writable MLog, and the paper states that if conflicting instances run in parallel, “MarlinCommit ensures that only one transaction is committed.” MigrationTxn spans source and destination ownership metadata. RecoveryMigrTxn is the clearest case: it may commit updates to both the source and destination GLogs even though the source node is unresponsive.

5. Guarantees, invariants, and non-claims

The paper states that “ACID properties are guaranteed for all transactions” and that MarlinCommit “ensures strong transactional guarantees even when multiple nodes can access the same data” (Hu et al., 3 Aug 2025). Atomicity follows from the protocol’s explicit atomic-commit structure: a multi-participant transaction either commits across all relevant participants or aborts. Durability follows from appending updates to durable, highly available storage-layer WALs; since logs are the “ground truth,” commit durability is log durability.

The treatment of isolation is more nuanced. The paper does not present a new formal isolation model specifically for MarlinCommit. Instead, it states that correctness of Marlin is independent of the isolation level of user transactions, that reconfiguration transactions are serializable, that user transactions must hold read locks on corresponding GTable entries until commit or abort, and that user-table accesses may use weaker isolation if desired. Ordinary local concurrency control remains responsible for ordinary transactional conflicts. MarlinCommit adds protection against cross-node metadata races that local CC cannot observe.

The paper summarizes correctness through invariants centered on granule ownership. I0: Exclusive Granule Ownership states that “for any granule $G$ at any time, there is exactly one owner node $N$.” Supporting invariants are I1: Reconfiguration Transactions are Serializable, I2: Node and GTable are 1-1 Mapped, I3: Owner Exists, and I4: Owner is Unique. The most direct MarlinCommit-related invariant is I1, whose operative meaning is that, for any log, transactions on that log are serialized via the TryLog operation.

A common overstatement is to treat this as a blanket linearizability claim. The paper does not explicitly claim that MarlinCommit provides linearizability as a general object-level property for all operations. What it does claim is strong consistency and serializability of reconfiguration transactions. The complete proof and TLA$^+$ specification are deferred to the appendix rather than developed fully in the main text.

6. Failure handling, stale-state recovery, and limitations

A major functional property of MarlinCommit is that transactions can progress even when compute nodes fail, because logs themselves can be participants. This is the mechanism by which RecoveryMigrTxn can update the failed source node’s GLog directly. The paper explicitly states that Marlin “does not have the blocking issue that traditional 2PC protocols suffer from,” and relates this to the same broad storage-disaggregated idea used by Cornus (Hu et al., 3 Aug 2025).

The paper gives a concrete stale-owner scenario. N2 suspects N3 has failed and runs RecoveryMigrTxn, moving granules G3 and G4 from N3 to N2. The transaction commits on both GLog2 and GLog3. N3 later recovers and tries a user transaction on G3. Its Append(updates, H-LSN) to GLog3 fails because it still tracks old H-LSN = 1, while GLog3 is now at 2. N3 then clears its GTable cache, reloads metadata, observes that it no longer owns G3 and G4, and aborts the transactions. The protocol thereby prevents stale-owner writes, double ownership, and lost-update style metadata races.

The same mechanism also addresses cases that can be described, in the paper’s terminology, as split brain on ownership, double commit on shared metadata, and orphan state. Only one append at the expected LSN can succeed, and ownership-change transactions swap ownership rather than delete it, aligning with the invariants “Owner Exists” and “Owner is Unique.”

The limitations are explicit. The paper does not provide a complete formal protocol state machine for all 2PC message states in the main text. Under high contention on MLog, performance can degrade because TryLog behaves like optimistic concurrency control with retries. Membership changes are assumed to be infrequent in practice, and the design is therefore optimized for the common case rather than a many-writer hotspot on a single shared metadata log.

7. Performance evidence and relation to prior work

The evaluation does not isolate MarlinCommit in a standalone microbenchmark; most results are system-level measurements for Marlin, especially migration and reconfiguration throughput and cost. Those results are nevertheless directly relevant because reconfiguration correctness and speed depend on MarlinCommit. In a YCSB scale-out with a 24 GB table, approximately 200K granules, 800 concurrent clients, scale-out from 8 to 16 nodes at 10s, and approximately 100K MigrationTxns, Marlin achieves 2.3× and 1.9× higher migration transaction throughput than small and large ZooKeeper deployments, completes scale-out 2.6× and 1.9× faster, reduces migration latency by 2.57× and 1.87×, and reduces cost by 1.35× and 1.61×. In TPC-C scale-out from 8 to 16 nodes with 6.4K MigrationTxns and 80 migration threads per new node, migration completes 2.5× and 1.5× faster than S-ZK and L-ZK, with less user-transaction disruption during migration. Across scale-outs from 1→2, 2→4, 4→8, and 8→16, Marlin attains the best reported cost/performance tradeoff, including up to 4.4× lower cost than L-ZK in SO1-2 and up to 2.1× lower cost per transaction than FoundationDB. In a geo-distributed setting, migration duration is up to 4.9× shorter than ZooKeeper-based approaches and up to 9.5× shorter than FoundationDB. In a dynamic workload with clients 400→800→400 and cluster 8→16→8, scale-out completes 2.6× and 2.3× faster than S-ZK and L-ZK, scale-in completes 3.8× and 2.6× faster, and scale-in releases nodes after 12s versus 45s and 32s for the ZooKeeper baselines (Hu et al., 3 Aug 2025).

The membership stress test is the clearest result tied specifically to MarlinCommit itself. Membership performance is comparable to ZooKeeper up to 160 nodes; beyond that, performance degrades because of optimistic-concurrency-control retry overhead in TryLog() on MLog. This corresponds directly to the protocol’s stated trade-off.

Relative to prior work, MarlinCommit differs from external converged coordination systems such as ZooKeeper, etcd, and Chubby by eliminating the external service and integrating coordination into the managed DBMS. It differs from sharded metadata services built atop FoundationDB by avoiding a separate external subsystem and by letting coordination scale with the database’s own scale. It differs from converged designs such as CockroachDB or KRaft-backed systems because Marlin assumes stateless, unreplicated compute nodes with durable access mediated through disaggregated storage rather than consensus-protected converged replicas. The paper identifies Cornus as the closest transactional comparison in spirit: Cornus removes the blocking problem of traditional 2PC in storage-disaggregated DBMSs by allowing an active node to commit to the log of an unresponsive participant. MarlinCommit adopts that broad idea but extends it with conditional append to detect cross-node modifications of shared coordination metadata.

In that sense, MarlinCommit is best understood as a storage-aware atomic commit protocol specialized for failover-critical metadata in a storage-disaggregated database: a transaction commits only if the relevant metadata logs remain at the versions the node believes they are, and otherwise aborts, invalidates stale metadata cache, refreshes, and retries.