DDMon: Distributed Deadlock Monitoring

• DDMon is a distributed deadlock monitoring framework that detects wait-for cycles in single-threaded RPC systems via a probe-chasing algorithm.
• The framework employs decentralized proxy monitors that observe message exchanges without modifying service code, ensuring transparency and precision.
• Formal guarantees of soundness and completeness are established through Coq mechanisation and rigorous empirical evaluations on Erlang/OTP applications.

DDMon, the Distributed Deadlock MONitoring framework, is a distributed black-box runtime approach for detecting deadlocks in concurrent and distributed RPC-based systems by deploying monitors alongside each service, observing only incoming and outgoing messages, and exchanging probes with other monitors (Rowicki et al., 20 Aug 2025). It is presented together with a formal model of SRPC (“single-threaded RPC”) services, a distributed black-box monitoring algorithm that is proved sound and complete—that is, it identifies deadlocked services with neither false positives nor false negatives—and an implementation for Erlang/OTP applications. The work also states that this is the first formalisation, correctness proof, and implementation of distributed black-box monitors for deadlock detection, and that the results are mechanised in Coq (Rowicki et al., 20 Aug 2025).

1. Problem setting and design objectives

DDMon is motivated by the structure of modern microservice and RPC-based systems in which individual services invoke each other in chains of remote procedure calls while each service is typically single-threaded, processing one request at a time and blocking on each RPC until its peer replies. In that setting, circular call chains can arise so that service $S_1$ waits for $S_2$, $S_2$ waits for $S_3$, and $S_n$ waits for $S_1$. Such wait-for cycles deadlock the involved services, and blocking propagates outward as other services stall waiting on the original victims (Rowicki et al., 20 Aug 2025).

The framework is designed for environments in which static analysis is inadequate because systems span language barriers, closed-source components, and heterogeneous frameworks. The data explicitly contrasts this with monolithic static analysis, which may be infeasible and must trade false positives for decidability. DDMon instead adopts a distributed black-box approach with four stated properties: it is distributed, so there is no central coordinator; black-box, because monitors only observe message sends and receives; transparent, because monitors do not change service behavior; and precise, because they detect deadlocks soundly and completely (Rowicki et al., 20 Aug 2025).

A central misconception that DDMon directly rejects is that accurate distributed deadlock detection necessarily requires source instrumentation, inlining, or a central bottleneck. The framework is explicitly presented as lightweight, black-box, and decentralized. Its objective is not approximate anomaly detection, but precise identification of real deadlocks in running systems.

2. SRPC model, locks, and deadlocks

The formal substrate of DDMon is the model of SRPC services, described as “single-threaded RPC” services and exemplified by Erlang/OTP gen_server or actor-based services. An SRPC service is written as

$\ser = \langle Q_\mathsf{in},\,P,\,Q_\mathsf{out}\rangle$

where $Q_\mathsf{in}$ and $Q_\mathsf{out}$ are FIFO-selective queues of messages $\msg{n}{\mathsf{q}/\mathsf{r}/\mathsf{c}}$, and $S_2$0 is the internal state (Rowicki et al., 20 Aug 2025).

The service discipline is sequential. In state $S_2$1, the process must dequeue a query or cast and move to $S_2$2. In $S_2$3, it may emit internal $S_2$4 actions, casts, and queries $S_2$5, then block in $S_2$6 until $S_2$7. After $S_2$8, it replies to client $S_2$9 unless $S_2$0 for casts, and returns to $S_2$1. This captures the gen_server or actor RPC pattern that the monitoring algorithm targets (Rowicki et al., 20 Aug 2025).

A network $S_2$2 is a mapping from service names to SRPC services. Its labeled transitions $S_2$3 are divided into two classes. Internal transitions, with $S_2$4, arise from a service’s internal $S_2$5, send, or receive steps. Communication transitions, with $S_2$6 for $S_2$7, denote a query or response sent by $S_2$8 and enqueued into $S_2$9. Paths are written $S_3$0 (Rowicki et al., 20 Aug 2025).

The deadlock semantics are likewise explicit. A service $S_3$1 is locked on $S_3$2 when its process state is $S_3$3, $S_3$4 is empty, and $S_3$5 lacks any pending $S_3$6. A deadlocked set $S_3$7 is a nonempty set of names such that each $S_3$8 is locked on some $S_3$9. A network has a deadlock if it contains any such set. The report further states, by Lemma 2.7, that once a deadlock arises it is persistent: no further execution can break it unless some blocked reply arrives, which cannot happen among the locked loop (Rowicki et al., 20 Aug 2025).

This persistence property is significant for DDMon because it turns deadlock detection into a problem of discovering an already-stable wait-for cycle rather than predicting a transient condition. The monitoring algorithm exploits exactly that stability.

3. Proxy monitors and the probe-chasing algorithm

DDMon instruments each service with a proxy monitor: $S_n$0 where $S_n$1 is the monitor’s internal state and $S_n$2 buffers both observed service messages and inter-monitor probes. Monitor behavior is driven by a total function

$S_n$3

Incoming service messages and probes are enqueued, then dequeued and fed to handle, which may emit zero or more outgoing probes $S_n$4 to other monitors (Rowicki et al., 20 Aug 2025).

The monitor state records three fields: probe, the current active probe $S_n$5 or ⊥; waiting, a set of peer names whose queries are currently in flight; and alarm, a Boolean flag. The operational logic is deliberately minimal. Whenever the monitored service sends a query to $S_n$6, the monitor draws a fresh probe $S_n$7 and sets probe := p. Whenever it receives a response, it sets probe := ⊥. Whenever it sees an incoming query $S_n$8, it records waiting ∪= {n}, and if probe ≠ ⊥, it immediately emits $S_n$9 (Rowicki et al., 20 Aug 2025).

Upon an incoming probe $S_1$0, the monitor performs one of three actions:

• Origin detection: if probe = p, then alarm := true.
• Silent drop: if probe = ⊥, the probe is dropped.
• Backward forwarding: otherwise, the probe is forwarded to every $S_1$1.

The report characterises this as backward-chasing along the SRPC “wait-for” edges. The essential handle equations are given as follows: $S_1$2 As soon as any monitor’s alarm flips to true, DDMon declares a deadlock and reports the concrete set of service names that forwarded the critical probe (Rowicki et al., 20 Aug 2025).

The importance of this construction is that it does not require monitors to inspect service code or reconstruct global control flow. Probe circulation is sufficient because the SRPC discipline constrains the shape of dependencies: a service is either waiting for a reply or not, and monitor-local knowledge of waiting and probe is enough to recover a deadlocked cycle.

4. Correctness criteria and Coq mechanisation

DDMon’s formal guarantees are split into two orthogonal criteria. The first is Transparency (Criterion 2.1), formulated as an operational correspondence between the original network and the instrumented one. Completeness of transparency means that every unmonitored step $S_1$3 lifts to some monitored path $S_1$4 with $S_1$5. Soundness of transparency means that monitored execution introduces no extra behaviors: every monitored path can be flushed by a few extra steps that simply deliver buffered messages back to some unmonitored step. The proofs proceed by induction on transition paths and use queue-filtering lemmas (Rowicki et al., 20 Aug 2025).

The second criterion is Detection Preciseness (Criterion 2.2). In any instrumented execution $S_1$6, the report states both:

• Soundness: if any monitor’s alarm=true, then the underlying unmonitored network $S_1$7 indeed contains a deadlocked set.
• Completeness: if $S_1$8 has a deadlock, then from $S_1$9 there is some finite continuation, obtained by flushing probes and exchanging messages, after which alarm must flip true (Rowicki et al., 20 Aug 2025).

Three invariants are identified as the main proof drivers: SRPC well-formedness, complete lock knowledge, and sound lock knowledge. SRPC well-formedness states that no service replies without a corresponding query and that mailbox and process state remain in sync. Complete lock knowledge states that no locked service ever goes un-probeed. Sound lock knowledge states that a non-⊥ probe always corresponds to a genuine lock (Rowicki et al., 20 Aug 2025).

The central combined result is Theorem 5.17: $\ser = \langle Q_\mathsf{in},\,P,\,Q_\mathsf{out}\rangle$0 An eventual-reporting refinement, Theorem 5.18, uses a lexicographic measure on pending probe-forwards together with a fairness assumption so that the report must actually occur in any fair run (Rowicki et al., 20 Aug 2025).

All semantic definitions, inductive invariants, and theorems are fully formalised in Coq 8.20. The mechanisation is structured into SRPC.v, Monitor.v, Algorithm.v, and Proofs.v. The report lists induction on LTS paths, custom Ltac inversion_path, simulation relations for operational correspondence, FSets-based finite set reasoning on waiting, and lexicographic measures using the standard Coq Compare module as key proof techniques. The mechanisation is associated with Zenodo 10.5281/zenodo.16909482, checks in under 5 KLOC, and scripts reproduce all proofs (Rowicki et al., 20 Aug 2025).

5. Erlang/OTP implementation and integration model

The prototype tool DDMon targets Erlang and Elixir applications using the OTP gen_server behaviour. Integration is presented as a drop-in replacement.

In Elixir: $\ser = \langle Q_\mathsf{in},\,P,\,Q_\mathsf{out}\rangle$1

In Erlang: $\ser = \langle Q_\mathsf{in},\,P,\,Q_\mathsf{out}\rangle$2

When DDMon starts, it spawns the real gen_server as a private child and returns the monitor PID to the application. All incoming requests first reach the monitor, which inspects them, enqueues them into its own mailbox, applies the handle logic to record waiting sets and emit probes, and then forwards them to the real service. Responses pass back through the monitor symmetrically. Probes are represented as tiny Erlang terms consisting of an integer tag plus the list of intermediates, which supports user-friendly reporting of the deadlocked set (Rowicki et al., 20 Aug 2025).

The implementation exposes deadlocks through concise logs such as: $\ser = \langle Q_\mathsf{in},\,P,\,Q_\mathsf{out}\rangle$3 and, in the usage section, $\ser = \langle Q_\mathsf{in},\,P,\,Q_\mathsf{out}\rangle$4 The reported set is described as precisely the services in the true wait-for cycle (Rowicki et al., 20 Aug 2025).

The operational integration procedure is also explicit: replace gen_server with ddmon, recompile without changing existing service code, optionally configure a probe delay via DDMon.set_probe_delay(1000) for a 1 s window, and run the application. The report states that DDMon may be toggled at runtime because it is fully transparent; removing DDMon restores the original message rates exactly (Rowicki et al., 20 Aug 2025).

6. Empirical evaluation, scope, and limitations

The evaluation uses synthetic benchmarks of randomly generated gen_server networks of up to 11 000 services. The reported overhead depends strongly on probe timing. With eager probe emission, service-to-service messages increase by approximately 100 %. With delayed emission using a 1 s timeout, that overhead drops to below 5 % for large systems because most probe candidates are cancelled by quick replies. When monitor-service forwarding is included, total messages triple, but each such send/receive occurs in RAM in the same Erlang VM and costs about 0.2 μs. Throughput benchmarks are said to confirm that, for typical workloads, end-to-end latency increases by under 2 % in the delayed setting. Under real traffic patterns, deadlocks are detected within tens of milliseconds after the cycle closes (Rowicki et al., 20 Aug 2025).

These measurements define the practical scope of DDMon. It is intended for SRPC patterns such as gen_server and gen_statem, where a black-box message-trace view is sufficient for exact reasoning about lock propagation. The framework does not yet support several cases listed in the report: AND-vs-OR RPC models, RPC timeouts that are invisible to pure message-trace monitoring unless services announce them to their proxy, a fully mechanised treatment of the fairness assumption behind eventual reporting, and cross-language deployments beyond the current Erlang-native implementation (Rowicki et al., 20 Aug 2025).

A plausible implication is that DDMon’s strongest guarantees depend on the alignment between the monitored runtime and the SRPC semantics formalised in the technical report. Within that scope, the framework combines distributed deployment, black-box observation, proved transparency, proved detection preciseness, Coq mechanisation, and a working Erlang/OTP implementation in a single system (Rowicki et al., 20 Aug 2025).