Distributed Backtrace (DBT) for RPC Debugging

• Distributed Backtrace (DBT) is a debugging mechanism that reconstructs unified call stacks across RPC boundaries using compact causality metadata.
• It leverages caller context capture and recursive upstream stack unwinding to enable interactive inspection of variables and heap objects in distributed systems.
• DBT integrates seamlessly into RPC frameworks with minimal code changes, offering low overhead and robust performance metrics in complex microservices environments.

Distributed Backtrace (DBT) is a source-level debugging mechanism for distributed applications that reconstructs a single, unified call stack across RPC boundaries. In DDB, DBT works by capturing caller context at each RPC send, embedding compact causality metadata in the RPC payload, and using that metadata to unwind upstream stacks on demand when execution is paused in a downstream service. The result is a backtrace that begins with local frames in the paused process and extends backward through remote caller processes, enabling interactive inspection of variables, arguments, and heap objects across service boundaries (Yan et al., 7 Jul 2026). In a broader observability context, aggregate trace analysis has also been proposed as a complementary way to group similar traces, select representative traces, and visualize canonical execution-path families, which suggests a useful comparative context for DBT-style diagnosis (Samanta et al., 2024).

1. Definition and conceptual boundaries

In conventional debuggers such as GDB and LLDB, a backtrace stops at the process boundary: if service $C$ is paused in an $A \rightarrow B \rightarrow C$ RPC chain, only $C$’s local stack is visible. DBT generalizes the notion of a call stack so that RPC sends and receives become part of one distributed stack. Its stated goals are to reconstruct causality across RPC boundaries, provide interactive source-level inspection across services, and remain robust to user-level threading, including goroutines and fibers that may migrate across OS threads (Yan et al., 7 Jul 2026).

This distinguishes DBT from distributed tracing and structured logging. Tracing systems such as Dapper, Zipkin, Jaeger, and OpenTelemetry reconstruct request paths and event histories, but they do not provide live source-level inspection of upstream callers. DBT, by contrast, is an interactive debugging mechanism: when execution pauses, a dbt command reconstructs the caller chain as one contiguous stack and exposes remote frames in a debugger UI (Yan et al., 7 Jul 2026).

A recurring misconception is to equate DBT with multi-process debugging in the HPC sense. Multi-process debuggers can coordinate breakpoints across processes, but DBT’s defining property is cross-RPC call-chain reconstruction, not merely simultaneous attachment. Another misconception is to conflate DBT with post-hoc trace exploration. Aggregate trace visualizations may summarize many requests, but DBT is concerned with a specific paused execution and its causal ancestry (Yan et al., 7 Jul 2026).

2. Causality metadata and distributed stack stitching

DBT rests on three design steps: capture caller context at the RPC call site, embed compact causality metadata in the RPC payload, and reconstruct a unified call stack when the application is paused. At the sender, DBT captures a process identifier, a thread context snapshot containing the register set needed to identify the caller’s stack, and a caller ID. The implementation explicitly does not rely on OS thread IDs alone; instead, it records register state so that stack reconstruction remains valid even if a user-level thread later moves to a different OS thread. In the reported evaluation, this metadata occupies 52 bytes per RPC (Yan et al., 7 Jul 2026).

On the receiving side, the RPC handler stores the extracted metadata in an injected frame, DDB::Backtrace::extraction, whose local variable meta becomes visible during unwinding. When dbt is invoked in a paused process, the local debugger agent unwinds the current stack using DWARF symbols, locates the injected extraction frame, reads meta, and obtains caller_id, caller_context, and caller process identity. The control plane then contacts the upstream process, temporarily restores the saved registers there, unwinds that caller stack, searches it for the next injected metadata frame, and repeats the procedure recursively until no valid metadata remains. The collected stacks are then concatenated in reverse request-path order, with callee frames first and root frames last (Yan et al., 7 Jul 2026).

The paper formalizes this as a DistributedBacktrace(thread_id) procedure built from FetchStack, ScanMetadata, GetCaller, RestoreStack, and StitchAllStacks. ABI-specific state restoration is required: for example, C++ on x86_64 uses RSP, RIP, and RBP, while aarch64 additionally uses lr. The stitched stack is represented as an ordered list of frames containing function, file, line, process ID, and thread ID. DBT does not introduce vector clocks or a general event DAG; it assumes tree-like request paths, or at least a linear RPC ancestry for the request that reached the paused process (Yan et al., 7 Jul 2026).

This design makes DBT “backtrace, but distributed” in a precise sense. The operative unit is not a trace span or a log line but a recoverable source-level stack frame. RPCs are treated as causal edges that permit stack continuation across address spaces.

3. Integration into DDB and operational environment

DBT is one of DDB’s three pillars, alongside the intent-preserving control plane and Pause-Erased Time (PET). The control plane manages debugger intents, propagates breakpoints and continue commands across dynamic process sets, and enforces pause-the-world semantics by default. DBT depends on it both to locate the correct upstream debugger agent for a given caller_process and to ensure that distributed stack reconstruction observes a consistent system state (Yan et al., 7 Jul 2026).

PET supplies the temporal isolation that makes DBT usable in real distributed systems. Its core definition is

$T_v(t) = T(t) - O(t),$

where $T(t)$ is kernel monotonic time and $O(t)$ is cumulative pause offset. By subtracting debugger-induced pauses from time observations, PET prevents timeout cascades, leader elections, crash recovery, and similar control-path perturbations that would otherwise destroy the debug context while DBT is traversing the distributed stack (Yan et al., 7 Jul 2026).

Integration occurs at the RPC framework layer rather than in application code. The reported modifications are approximately 20 LOC for gRPC C++, 30 LOC for Nu, 60 LOC for Quicksand, and 10 LOC for ServiceWeaver. On the client side, the framework hooks the RPC send path, invokes a connector to build meta from the current thread, and serializes that metadata into the payload. On the server side, it extracts the metadata, pushes the injected extraction frame, and stores meta as a local variable visible to unwinding. After this one-time framework integration, applications that use the framework gain DBT without source changes in user code (Yan et al., 7 Jul 2026).

The same paper places DBT in several concrete environments. In gRPC Raft, a breakpoint in a follower’s AppendEntries handler combined with dbt reveals the leader’s upstream send_heartbeat frame and its arguments. In ServiceWeaver socialnet, described as 36 microservices with 5 replicas each, DBT allows debugging across a large process set without manually attaching to every instance. In Nu and Quicksand, where computation migration and memory disaggregation are present, DBT metadata follows logical computations, and a plugin can restore migrated heap state so that remote frames continue to expose heap variables correctly (Yan et al., 7 Jul 2026).

4. Aggregate trace analysis as a comparative layer

A separate but closely related line of work addresses a limitation that DBT does not itself solve: understanding the typical or canonical execution pattern of a request class across large trace corpora. “Visualizing Distributed Traces in Aggregate” proposes preprocessing to remove incomplete traces when a more complete version exists, grouping similar traces based on service sets or edge sets, selecting a representative trace for each group, and constructing an aggregate trace data structure that summarizes service and call frequencies within that group (Samanta et al., 2024).

The formal machinery is set-based and graph-based. Traces are encoded either as service sets $S(T)$ or edge sets $E(T)$, and similarity is measured with Jaccard indices such as

$$J(T_i,T_j)=\frac{|S(T_i)\cap S(T_j)|}{|S(T_i)\cup S(T_j)|}$$

or its structural analogue over edge sets. Given a threshold $\theta$, a trace-similarity graph is formed and grouped via Disjoint Set Union. For a group $A \rightarrow B \rightarrow C$0, the representative trace is

$A \rightarrow B \rightarrow C$1

that is, the trace most connected to the others in the induced similarity subgraph (Samanta et al., 2024).

Before grouping, the paper filters incomplete traces using a strict-subgraph criterion: $A \rightarrow B \rightarrow C$2 If one trace’s edge set is a strict subset of another’s, the shorter trace is treated as incomplete and removed. The authors explicitly note a limitation: some shorter traces may be complete workflows, such as cache-hit or early-exit cases, rather than truncated observations (Samanta et al., 2024).

The aggregate structure for each group is a weighted service-dependency graph

$A \rightarrow B \rightarrow C$3

where $A \rightarrow B \rightarrow C$4 is the group’s service set, $A \rightarrow B \rightarrow C$5 is the number of traces, $A \rightarrow B \rightarrow C$6 counts how many traces contain service $A \rightarrow B \rightarrow C$7, and $A \rightarrow B \rightarrow C$8 counts service-to-service calls across the group. Visualization uses graph-tool: nodes appearing in all traces of the group are yellow, nodes appearing only in some traces are gray, node size is proportional to occurrence count, and one chosen service can be highlighted in green with outgoing edge thickness proportional to call frequency (Samanta et al., 2024).

This does not implement DBT in the DDB sense, but it suggests a natural companion abstraction. A representative trace can serve as a canonical distributed backtrace for a request family, and the aggregate graph can show mandatory versus optional steps around that canonical chain. In that interpretation, DBT supplies interactive per-request causal reconstruction, while aggregate trace analysis supplies the “normal form” against which an anomalous backtrace can be compared.

5. Empirical behavior and debugging workflows

DDB evaluates DBT on gRPC, ServiceWeaver, Nu, and Quicksand across up to 122 processes. The abstract summarizes the system as achieving 30ms median cross-RPC backtrace latency, sub-5 ms time jump under repeated execution pauses, and 1–5% throughput overhead, while the detailed scalability experiment reports median dbt latencies of 14.2 ms at 38 pods, 13.9 ms at 62 pods, and 13.3 ms at 122 pods. Tail latencies remain under approximately 160 ms even when thousands of concurrent dbt requests are issued, as happens when an IDE requests stacks for every thread (Yan et al., 7 Jul 2026).

Latency grows with distributed call depth. The reported median latency is 48.0 ms at depth 2, 120.9 ms at depth 3, 173.0 ms at depth 4, 216.5 ms at depth 5, 259.0 ms at depth 6, and 434.9 ms at depth 10. The per-RPC metadata cost is 52 bytes. Throughput impact is reported as a 0.6% drop for ServiceWeaver socialnet and a 3.8% drop for Nu socialnet; in gRPC Raft, DDB as a whole adds approximately 20.1% overhead, compared to approximately 18.4% for attaching plain GDB, which the paper attributes to debugger overhead rather than DBT specifically (Yan et al., 7 Jul 2026).

The controlled user study is central to the practical argument for DBT. DDB reports a 100% fault-localization success rate, compared to 38.5% for baseline tools, with a median localization time of approximately 8 minutes. The paper’s representative debugging scenario is Raft heartbeat rejection: a breakpoint in the follower’s handler, global pause via the control plane, dbt to surface the leader’s send_heartbeat frame, and then inspection of leader term, request arguments, and follower state in one debugger session. Another scenario is a microservice chain in socialnet, where a downstream failure may actually originate in upstream routing or serialization; dbt exposes that upstream state directly rather than requiring manual correlation across logs or traces (Yan et al., 7 Jul 2026).

These workflows reveal the distinctive operational niche of DBT. It is most valuable when failure manifestation and fault origin lie in different services, and when the developer needs source-level state rather than aggregate observability. In that respect, DBT is a debugging primitive rather than a monitoring primitive.

6. Limitations, related paradigms, and acronym ambiguity

DBT, as implemented in DDB, is explicitly tied to RPC-based causality. It does not cover arbitrary asynchronous message passing or shared-memory concurrency bugs. It requires RPC-layer integration, so the reconstructed backtrace stops at frameworks that have not been instrumented. If an upstream caller has terminated before the callee pauses, DBT returns a truncated backtrace and reports “caller terminated.” PET only virtualizes time inside the attached cluster, so external dependencies with strict physical-time behavior may still time out during debugging. The prototype also depends on GDB and DWARF, and ABI-specific register-restore logic is required for each supported platform (Yan et al., 7 Jul 2026).

The main scaling trade-off is depth and ancestry representation. The paper notes that backtrace latency scales with call depth, and suggests that carrying more ancestors in metadata could reduce latency at the cost of larger payloads. Aggregate-trace work identifies a different scaling problem: pairwise similarity computation is $A \rightarrow B \rightarrow C$9, which becomes problematic for millions of traces. That work therefore suggests indexing, approximate similarity such as MinHash, and larger-scale experiments on DeathStarBench as future directions (Samanta et al., 2024).

A related but distinct debugging paradigm appears in URDB, which uses DMTCP checkpoint-restart to support reversible debugging and temporal search for multi-threaded, multi-process, and distributed computations. URDB can synchronously roll back all participating processes and consistently restore communication state, and this suggests an alternative substrate for distributed backtrace systems centered on temporal navigation and reverse watchpoints rather than RPC-spanning stack stitching (0910.5046). The suggestion is architectural rather than terminological: URDB does not define DBT as a unified cross-RPC call stack.

The acronym itself is overloaded. In software testing, DBT refers to Diversity-Based Testing, a family of techniques that use similarity metrics over artefacts to generate, prioritize, select, reduce, or evaluate tests (Elgendy et al., 2023). In imaging, DBT refers to digital breast tomosynthesis, as in work on PDHG-based image reconstruction for wide-angle DBT (Sidky et al., 24 Apr 2026). At the network layer, traceback research studies algebraic reconstruction of router paths in dynamic networks, including incremental update schemes using $C$0 marked packets for initial path discovery and $C$1 marked packets for subsequent topology changes, but that notion of traceback concerns packet routes rather than source-level distributed call stacks (0908.0078). In current distributed debugging literature, however, Distributed Backtrace most precisely denotes the cross-RPC stack-reconstruction mechanism introduced in DDB (Yan et al., 7 Jul 2026).