---
title: 'DDMon: Distributed Deadlock Monitoring'
url: https://www.emergentmind.com/topics/ddmon
type: topic
---

# DDMon: Distributed Deadlock Monitoring

DDMon, the **Distributed Deadlock MONitoring** framework, is a distributed black-box runtime approach for detecting deadlocks in concurrent and distributed RPC-based systems by deploying monitors alongside each service, observing only incoming and outgoing messages, and exchanging probes with other monitors [2508.14851]. It is presented together with a formal model of **SRPC** (“single-threaded RPC”) services, a distributed black-box monitoring algorithm that is proved **sound and complete**—that is, it identifies deadlocked services with neither false positives nor false negatives—and an implementation for Erlang/OTP applications. The work also states that this is the first formalisation, correctness proof, and implementation of distributed black-box monitors for deadlock detection, and that the results are mechanised in Coq [2508.14851].

## 1. Problem setting and design objectives

DDMon is motivated by the structure of modern microservice and RPC-based systems in which individual services invoke each other in chains of remote procedure calls while each service is typically single-threaded, processing one request at a time and blocking on each RPC until its peer replies. In that setting, circular call chains can arise so that service \(S_1\) waits for \(S_2\), \(S_2\) waits for \(S_3\), and \(S_n\) waits for \(S_1\). Such **wait-for cycles** deadlock the involved services, and blocking propagates outward as other services stall waiting on the original victims [2508.14851].

The framework is designed for environments in which static analysis is inadequate because systems span language barriers, closed-source components, and heterogeneous frameworks. The data explicitly contrasts this with monolithic static analysis, which may be infeasible and must trade false positives for decidability. DDMon instead adopts a **distributed black-box** approach with four stated properties: it is **distributed**, so there is no central coordinator; **black-box**, because monitors only observe message sends and receives; **transparent**, because monitors do not change service behavior; and **precise**, because they detect deadlocks soundly and completely [2508.14851].

A central misconception that DDMon directly rejects is that accurate distributed deadlock detection necessarily requires source instrumentation, inlining, or a central bottleneck. The framework is explicitly presented as lightweight, black-box, and decentralized. Its objective is not approximate anomaly detection, but precise identification of real deadlocks in running systems.

## 2. SRPC model, locks, and deadlocks

The formal substrate of DDMon is the model of **SRPC** services, described as “single-threaded RPC” services and exemplified by Erlang/OTP `gen_server` or actor-based services. An SRPC service is written as
\[
\ser = \langle Q_\mathsf{in},\,P,\,Q_\mathsf{out}\rangle
\]
where \(Q_\mathsf{in}\) and \(Q_\mathsf{out}\) are FIFO-selective queues of messages \(\msg{n}{\mathsf{q}/\mathsf{r}/\mathsf{c}}\), and \(P\) is the internal state [2508.14851].

The service discipline is sequential. In state \(\mathsf{ready}\), the process must dequeue a query or cast and move to \(\mathsf{working}(c)\). In \(\mathsf{working}(c)\), it may emit internal \(\tau\) actions, casts, and queries \(\sendq{s}\), then block in \(\mathsf{locked}(c,s)\) until \(\recvr{s}\). After \(\recvr{s}\), it replies to client \(c\) unless \(c=\bot\) for casts, and returns to \(\mathsf{ready}\). This captures the gen_server or actor RPC pattern that the monitoring algorithm targets [2508.14851].

A network \(\Net\) is a mapping from service names to SRPC services. Its labeled transitions \(\Net \xrightarrow{\alpha} \Net'\) are divided into two classes. **Internal** transitions, with \(\alpha=\tau\), arise from a service’s internal \(\tau\), send, or receive steps. **Communication** transitions, with \(\alpha=n\!\rightarrow m:t\) for \(t\in\{\mathsf{q},\mathsf{r}\}\), denote a query or response sent by \(n\) and enqueued into \(m\). Paths are written \(\Net \xRightarrow{\pi} \Net'\) [2508.14851].

The deadlock semantics are likewise explicit. A service \(\ser\) is **locked** on \(s\) when its process state is \(\mathsf{locked}(c,s)\), \(Q_\mathsf{out}\) is empty, and \(Q_\mathsf{in}\) lacks any pending \(\recvr{s}\). A **deadlocked set** \(D\) is a nonempty set of names such that each \(\name\in D\) is locked on some \(s\in D\). A network has a deadlock if it contains any such set. The report further states, by Lemma 2.7, that once a deadlock arises it is **persistent**: no further execution can break it unless some blocked reply arrives, which cannot happen among the locked loop [2508.14851].

This persistence property is significant for DDMon because it turns deadlock detection into a problem of discovering an already-stable wait-for cycle rather than predicting a transient condition. The monitoring algorithm exploits exactly that stability.

## 3. Proxy monitors and the probe-chasing algorithm

DDMon instruments each service with a **proxy monitor**:
\[
\mser = \langle \mathit{MQueue},\,M,\,\ser\rangle,
\]
where \(M\) is the monitor’s internal state and \(\mathit{MQueue}\) buffers both observed service messages and inter-monitor probes. Monitor behavior is driven by a total function
\[
\mathit{handle}:\,M\times \mathit{MQueueElem}\;\to\; M\times \mathit{MQueue}.
\]
Incoming service messages and probes are enqueued, then dequeued and fed to `handle`, which may emit zero or more outgoing probes \(\msend{n}{p}\) to other monitors [2508.14851].

The monitor state records three fields: `probe`, the current active probe \(p\) or `⊥`; `waiting`, a set of peer names whose queries are currently in flight; and `alarm`, a Boolean flag. The operational logic is deliberately minimal. Whenever the monitored service **sends** a query to \(n\), the monitor draws a fresh probe \(p\) and sets `probe := p`. Whenever it **receives** a response, it sets `probe := ⊥`. Whenever it **sees** an incoming query \(\recvq{n}\), it records `waiting ∪= {n}`, and if `probe ≠ ⊥`, it immediately emits \(\msend{n}{probe}\) [2508.14851].

Upon an incoming probe \(\mrecv{n}{p}\), the monitor performs one of three actions:

- **Origin detection**: if `probe = p`, then `alarm := true`.
- **Silent drop**: if `probe = ⊥`, the probe is dropped.
- **Backward forwarding**: otherwise, the probe is forwarded to every \(m \in waiting\).

The report characterises this as **backward-chasing** along the SRPC “wait-for” edges. The essential `handle` equations are given as follows:
\[
\begin{array}{l}
\handle(M,\sendq{n}) \;=\;(M[\mathit{probe}\mapsto p_\mathit{new}],\,\varepsilon)\\
\handle(M,\recvr{n}) \;=\;(M[\mathit{probe}\mapsto\bot],\,\varepsilon)\\
\handle(M,\recvq{n}):~
\begin{cases}
M.\mathit{waiting}\mathrel{:=}M.\mathit{waiting}\cup\{n\};~\varepsilon &\text{if }M.\mathit{probe}=\bot,\\
M.\mathit{waiting}\mathrel{:=}M.\mathit{waiting}\cup\{n\};~[\msend{n}{M.\mathit{probe}}] &\text{otherwise,}
\end{cases}\\
\handle(M,\mrecv{n}{p}):~
\begin{cases}
(M[\mathit{alarm}\mapsto\mathsf{true}],\,\varepsilon) &\text{if }M.\mathit{probe}=p,\\
(M,\varepsilon) &\text{if }M.\mathit{probe}=\bot,\\
(M,\,\{\msend{m}{p}\mid m\in M.\mathit{waiting}\}) &\text{otherwise.}
\end{cases}
\end{array}
\]
As soon as any monitor’s `alarm` flips to true, DDMon declares a deadlock and reports the concrete set of service names that forwarded the critical probe [2508.14851].

The importance of this construction is that it does not require monitors to inspect service code or reconstruct global control flow. Probe circulation is sufficient because the SRPC discipline constrains the shape of dependencies: a service is either waiting for a reply or not, and monitor-local knowledge of `waiting` and `probe` is enough to recover a deadlocked cycle.

## 4. Correctness criteria and Coq mechanisation

DDMon’s formal guarantees are split into two orthogonal criteria. The first is **Transparency (Criterion 2.1)**, formulated as an **operational correspondence** between the original network and the instrumented one. Completeness of transparency means that every unmonitored step \(\Net\to\Net'\) lifts to some monitored path \(\instr(\Net)\to\MNet'\) with \(\deinstr(\MNet')=\Net'\). Soundness of transparency means that monitored execution introduces no extra behaviors: every monitored path can be flushed by a few extra steps that simply deliver buffered messages back to some unmonitored step. The proofs proceed by induction on transition paths and use queue-filtering lemmas [2508.14851].

The second criterion is **Detection Preciseness (Criterion 2.2)**. In any instrumented execution \(\instr(\Net)\overset{\pi}{\Longrightarrow}\MNet\), the report states both:
- **Soundness**: if any monitor’s `alarm=true`, then the underlying unmonitored network \(\deinstr(\MNet)\) indeed contains a deadlocked set.
- **Completeness**: if \(\deinstr(\MNet)\) has a deadlock, then from \(\MNet\) there is some finite continuation, obtained by flushing probes and exchanging messages, after which `alarm` must flip true [2508.14851].

Three invariants are identified as the main proof drivers: **SRPC well-formedness**, **complete lock knowledge**, and **sound lock knowledge**. SRPC well-formedness states that no service replies without a corresponding query and that mailbox and process state remain in sync. Complete lock knowledge states that no locked service ever goes un-`probe`ed. Sound lock knowledge states that a non-`⊥` `probe` always corresponds to a genuine lock [2508.14851].

The central combined result is Theorem 5.17:
\[
\begin{array}{l}
\text{Let }\Net\text{ be initial SRPC network and }\instr\text{ its DDMon instrumentation.}\\
\forall\,\MNet,\pi:\instr(\Net)\!\xRightarrow{\pi}\!\MNet: \\
\quad\text{(a) If }\exists\,D\text{ deadlock in }\deinstr(\MNet), \text{ then } \MNet\to^*\MNet'\text{ with a report.}\\
\quad\text{(b) If }\MNet\text{ reports a deadlock, then }D\text{ exists in }\deinstr(\MNet).
\end{array}
\]
An eventual-reporting refinement, Theorem 5.18, uses a lexicographic measure on pending probe-forwards together with a fairness assumption so that the report must actually occur in any fair run [2508.14851].

All semantic definitions, inductive invariants, and theorems are fully formalised in **Coq 8.20**. The mechanisation is structured into `SRPC.v`, `Monitor.v`, `Algorithm.v`, and `Proofs.v`. The report lists induction on LTS paths, custom Ltac `inversion_path`, simulation relations for operational correspondence, FSets-based finite set reasoning on `waiting`, and lexicographic measures using the standard Coq `Compare` module as key proof techniques. The mechanisation is associated with Zenodo `10.5281/zenodo.16909482`, checks in under **5 KLOC**, and scripts reproduce all proofs [2508.14851].

## 5. Erlang/OTP implementation and integration model

The prototype tool **DDMon** targets Erlang and Elixir applications using the OTP `gen_server` behaviour. Integration is presented as a **drop-in replacement**.

In Elixir:
```elixir
alias :ddmon, as: GenServer
```

In Erlang:
```erlang
-compile({parse_transform, ddmon_transform}).
```

When `DDMon` starts, it spawns the real `gen_server` as a private child and returns the **monitor PID** to the application. All incoming requests first reach the monitor, which inspects them, enqueues them into its own mailbox, applies the `handle` logic to record waiting sets and emit probes, and then forwards them to the real service. Responses pass back through the monitor symmetrically. Probes are represented as tiny Erlang terms consisting of an integer tag plus the list of intermediates, which supports user-friendly reporting of the deadlocked set [2508.14851].

The implementation exposes deadlocks through concise logs such as:
```text
[monitor@host] DEADLOCK detected: services = {s1,s2,s3}
```
and, in the usage section,
```text
[ddmon] Deadlock among {svcA,svcB,svcC}
```
The reported set is described as precisely the services in the true wait-for cycle [2508.14851].

The operational integration procedure is also explicit: replace `gen_server` with `ddmon`, recompile without changing existing service code, optionally configure a probe delay via `DDMon.set_probe_delay(1000)` for a 1 s window, and run the application. The report states that DDMon may be toggled at runtime because it is fully transparent; removing DDMon restores the original message rates exactly [2508.14851].

## 6. Empirical evaluation, scope, and limitations

The evaluation uses synthetic benchmarks of randomly generated `gen_server` networks of up to **11 000** services. The reported overhead depends strongly on probe timing. With **eager** probe emission, service-to-service messages increase by approximately **100 %**. With **delayed** emission using a **1 s timeout**, that overhead drops to below **5 %** for large systems because most probe candidates are cancelled by quick replies. When monitor-service forwarding is included, total messages triple, but each such send/receive occurs in RAM in the same Erlang VM and costs about **0.2 μs**. Throughput benchmarks are said to confirm that, for typical workloads, end-to-end latency increases by under **2 %** in the delayed setting. Under real traffic patterns, deadlocks are detected within **tens of milliseconds** after the cycle closes [2508.14851].

These measurements define the practical scope of DDMon. It is intended for **SRPC** patterns such as `gen_server` and `gen_statem`, where a black-box message-trace view is sufficient for exact reasoning about lock propagation. The framework does **not yet support** several cases listed in the report: **AND-vs-OR RPC models**, **RPC timeouts** that are invisible to pure message-trace monitoring unless services announce them to their proxy, a fully mechanised treatment of the **fairness** assumption behind eventual reporting, and **cross-language deployments** beyond the current Erlang-native implementation [2508.14851].

A plausible implication is that DDMon’s strongest guarantees depend on the alignment between the monitored runtime and the SRPC semantics formalised in the technical report. Within that scope, the framework combines distributed deployment, black-box observation, proved transparency, proved detection preciseness, Coq mechanisation, and a working Erlang/OTP implementation in a single system [2508.14851].

Source: https://www.emergentmind.com/topics/ddmon