ALock: Asymmetric Hierarchical Locks

• Asymmetric Hierarchical Locks (ALock) are a hierarchical locking primitive for RDMA systems that use local and remote cohorts for efficient mutual exclusion.
• It eliminates RDMA loopback and RPC overheads by leveraging embedded MCS queue locks and a global Peterson-style arbitration for fast, fair access.
• Experimental evaluations show ALock achieves up to 29× throughput and 20× latency improvements over traditional RDMA locks under locality-dominant workloads.

Asymmetric Hierarchical Locks (ALock) are a locking primitive specifically designed for RDMA-based (Remote Direct Memory Access) systems, enabling efficient synchronization between local and remote threads accessing shared memory regions. ALock achieves mutual exclusion and starvation-freedom while eliminating RDMA loopback and remote procedure call (RPC) overheads, outperforming traditional RDMA locks in high-locality and mixed-locality workloads. The protocol leverages a two-cohort hierarchy—local and remote—embedded MCS queue locks, and a global Peterson-style arbitration to provide both performance and correctness guarantees (Baran et al., 2024).

1. Formal Specification

An ALock instance maintains state over two cohorts, local ($L$) and remote ($R$), each managed by an MCS queue. The protocol state is

$\mathsf{ALock} = (\tail_r, \; \tail_\ell, \; \victim, \; \{\mathit{desc}_p\}_{p\in P}),$

where $\tail_r$ and $\tail_\ell$ denote the tails of the remote and local cohorts (with zero indicating no waiters), $\victim \in \{\mathrm{R},\mathrm{L}\}$ is the Peterson-style global arbitrator, and $$\mathit{desc}_p = (\mathit{next}_p, \mathit{budget}_p)$$ is the descriptor for each process $p$. Each process $p$ tracks a Boolean $\mathit{passed}_p$ to indicate if it acquired the lock by passing through the cohort or by reacquire.

The locking procedure operates as follows:

• Each operation is dispatched to the respective cohort queue (local or remote) via the $R$0 function.
• Cohorts arbitrate global lock ownership through a Peterson algorithm: $R$1, guaranteeing exclusive progress for one cohort.
• State transitions $R$2, $R$3, and $R$4 are responsible for MCS-style acquisition, release, and Peterson-level reacquisition, respectively.

Safety (mutual exclusion) is specified as

$R$5

and liveness (starvation-freedom) is guaranteed by the budgeted reacquire logic and fair atomic primitives, formally: $R$6

2. Locking and Unlocking Procedures

The ALock protocol's pseudocode operates in three key modules: the high-level lock/unlock entrypoints, the modified per-cohort MCS queues, and the global Peterson arbitration invoked on budget expiration or missed fast paths.

Lock Acquisition:

$\tail_r$3

Unlock:

$\tail_r$4

Per-cohort MCS Lock Acquisition (qLock_c) and Release (qUnLock_c): Both operations use local or RDMA-native (remote) CAS and write primitives, depending on cohort type. Fast-path shared-memory passes are enabled for 100% locality cases, with only rare global Peterson reacquire required ($R$7 rate).

Peterson Reacquire: Upon budget expiry or initial acquisition, $R$8 synchronizes access between the cohorts, ensuring global fairness.

3. Theoretical Performance Analysis

Define $R$9 as local atomic primitive latency, $\mathsf{ALock} = (\tail_r, \; \tail_\ell, \; \victim, \; \{\mathit{desc}_p\}_{p\in P}),$0 as RDMA atomic latency, and $\mathsf{ALock} = (\tail_r, \; \tail_\ell, \; \victim, \; \{\mathit{desc}_p\}_{p\in P}),$1 as the spin-wait cost (negligible).

Worst-case hold-time:

$\mathsf{ALock} = (\tail_r, \; \tail_\ell, \; \victim, \; \{\mathit{desc}_p\}_{p\in P}),$2

This path includes one remote cohort-acquire, one remote cohort-release, plus Peterson reacquire on budget expiration.

Average-case hold-time:

$\mathsf{ALock} = (\tail_r, \; \tail_\ell, \; \victim, \; \{\mathit{desc}_p\}_{p\in P}),$3

Most requests are local fast-path passes, incurring only local memory operations. Only an $\mathsf{ALock} = (\tail_r, \; \tail_\ell, \; \victim, \; \{\mathit{desc}_p\}_{p\in P}),$4 fraction require re-entry to the global Peterson layer, incurring additional $\mathsf{ALock} = (\tail_r, \; \tail_\ell, \; \victim, \; \{\mathit{desc}_p\}_{p\in P}),$5 overhead.

Empirical latencies: $\mathsf{ALock} = (\tail_r, \; \tail_\ell, \; \victim, \; \{\mathit{desc}_p\}_{p\in P}),$6 Thus, in locality-dominated conditions, the hierarchical split yields up to $\mathsf{ALock} = (\tail_r, \; \tail_\ell, \; \victim, \; \{\mathit{desc}_p\}_{p\in P}),$7 speedup over pure RDMA locks.

4. Experimental Evaluation

ALock was evaluated on a distributed lock table deployed across 5–20 CloudLab nodes (Intel Xeon E5, Mellanox CX-3). The following table summarizes throughput (lock+unlock ops/sec) and 95th percentile tail-latency for ALock versus MCS-RDMA and RDMA spinlock competitors under various workloads:

Workload	ALock	MCS-RDMA	Spinlock-RDMA	Relative Speedup (ALock/MSCS)
100% locality, 20 locks, 240 threads (high contention)	1.2M ops/s	50k ops/s	55k ops/s	24×
95% locality, 100 locks, 80 threads (medium contention)	880k ops/s	340k ops/s	290k ops/s	2.6×
1000 locks, low contention	1.5M ops/s	900k ops/s	450k ops/s	1.7×–3.3×
Tail-latency (95%), 100% locality	120 ns	2.1 μs	3.9 μs	—

With increasing locality, ALock's throughput and latency approach those of pure shared-memory locks. As contention or remote operation fractions increase, ALock demonstrates up to 29× throughput and 20× latency improvements under certain settings.

5. Protocol Advantages

• Elimination of RDMA loopback: For purely local acquisitions, all operations use shared-memory CAS rather than going through the RDMA NIC, yielding minimum latency ($\mathsf{ALock} = (\tail_r, \; \tail_\ell, \; \victim, \; \{\mathit{desc}_p\}_{p\in P}),$850 ns).
• Starvation-freedom and fairness: Budgeted Peterson reacquire ensures all processes make progress under high contention.
• Local spinning and precise hand-off: Favouring per-cohort MCS queues avoids RNIC congestion and supports strict FIFO fairness.
• Connection reduction: For local nodes, the protocol reduces the number of required RDMA connections by $\mathsf{ALock} = (\tail_r, \; \tail_\ell, \; \victim, \; \{\mathit{desc}_p\}_{p\in P}),$9, mitigating QP thrashing.
• Formal verification: A TLA+–verified model and open-source implementation are provided, offering strong correctness guarantees.

6. Limitations and Integration Considerations

• Budget tuning: Cohort budgets $\tail_r$0 must be tuned to match workload characteristics; defaults ($\tail_r$1) were empirically validated.
• Budget expiration overhead: Occasional global reacquire incurs $\tail_r$2 cost.
• Descriptor management: The protocol incurs the overhead of per-thread descriptor allocation and pointer-chasing.
• Two-cohort limitation: The present design manages only two cohorts; generalizing beyond local/remote access patterns requires extension.
• Memory alignment and RDMA constraints: Metadata (two 8-byte tails, one 4-byte victim, padding) should be 64-byte aligned; pointer encodings must encode home-node for locality. Fencing and ordering for atomic operations must respect RDMA memory model semantics.
• NIC resource consideration: The protocol is practical on standard ConnectX-3/4 NICs but requires caution to prevent QP cache thrashing in very large deployments.

7. Relation to RDMA Synchronization Landscape

ALock bridges the architectural gap between local-only mutual exclusion and global, RDMA-driven locks. The protocol eliminates slow loopback or RPC fallbacks and provides substantial throughput (up to 29×) and latency (<1/20th) reductions in locality-heavy high-concurrency environments (Baran et al., 2024). Its hierarchical split, combining local fast-paths and global fairness mechanisms, addresses both scalability and performance symmetry, marking a significant step in the evolution of synchronization primitives for high-performance distributed systems.