CrossTrace: Tracing & Reasoning Insights

• CrossTrace is a shared title representing two distinct works: an eBPF-based zero-code tracing system for microservices and a dataset of grounded scientific reasoning traces.
• In distributed tracing, the system uses greedy delay-pattern inference and TCP option header embedding to correlate spans accurately without source-code modifications.
• The dataset provides 1,389 step-level reasoning traces across domains, enhancing hypothesis generation with verifiable, grounded quotations from source literature.

CrossTrace is a shared title used for two unrelated arXiv contributions. In distributed systems, "CrossTrace: Efficient Cross-Thread and Cross-Service Span Correlation in Distributed Tracing for Microservices" denotes a zero-code, eBPF-based tracing system for microservice observability that addresses intra-service and inter-service span correlation without source-code modification (Phan et al., 15 Aug 2025). In scientific reasoning, "CrossTrace: A Cross-Domain Dataset of Grounded Scientific Reasoning Traces for Hypothesis Generation" denotes a dataset of 1,389 grounded reasoning traces spanning biomedical research, AI/ML, and cross-domain work, together with an Input/Trace/Output schema and fine-tuning results for hypothesis generation (Bouras et al., 30 Mar 2026). The name therefore functions as a homonym rather than a single research program.

1. Disambiguation and nomenclature

The two CrossTrace works occupy different technical domains and solve different problems. One concerns distributed tracing for microservices; the other concerns grounded scientific reasoning traces for LLM training and evaluation. A related but distinct acronym, TRACE, expands to "Trajectory Correction from Cross-layer Evidence for Hallucination Reduction" and addresses hallucination correction rather than either microservice observability or hypothesis-generation datasets (Ranade, 18 May 2026).

Work	Domain	Core contribution
CrossTrace (Phan et al., 15 Aug 2025)	Distributed tracing for microservices	eBPF-based zero-code tracing with greedy delay-based intra-service correlation and TCP-option span ID propagation
CrossTrace (Bouras et al., 30 Mar 2026)	Grounded scientific reasoning / hypothesis generation	Dataset of 1,389 grounded reasoning traces with Input/Trace/Output schema and QLoRA fine-tuning on Qwen2.5-7B-Instruct
TRACE (Ranade, 18 May 2026)	Hallucination reduction in LLMs	Deterministic, training-free correction from cross-layer candidate trajectories

A common misconception is that CrossTrace refers to one framework with multiple modules. The cited record instead shows two separate papers with the same title string. Another possible confusion arises from the lexical similarity between CrossTrace and TRACE; the latter is a separate method with a different acronym expansion and problem setting.

2. CrossTrace as a distributed tracing system for microservices

In the microservice literature, CrossTrace is motivated by the burden of manual instrumentation in traditional, code-centric tracing, including OpenTelemetry SDKs, and by the limitations of zero-code alternatives that merely capture spans without reliably correlating them into end-to-end traces (Phan et al., 15 Aug 2025). The paper frames span correlation as the critical problem: capturing ingress and egress timestamps is insufficient unless parent-child and cross-service relationships can be reconstructed.

The system is positioned against three categories of prior zero-code solutions. Thread-affinity approaches such as DeepFlow and Grafana Beyla assume that ingress and egress spans on the same OS thread form a parent-child pair; the paper states that this assumption fails in systems that multiplex requests over virtual threads or worker pools, including Go goroutines and Java virtual threads. Grafana Beyla’s header-injection approach is also described as requiring kernel integrity or lockdown to be disabled, which the paper characterizes as unacceptable in production due to security policies. TraceWeaver avoids thread identifiers, but relies on a global optimization step whose runtime grows super-linearly under high concurrency.

CrossTrace addresses these constraints through two design choices. For intra-service correlation, it replaces thread-ID matching with a greedy delay-pattern inference algorithm. For inter-service correlation, it embeds span identifiers into TCP option headers via eBPF rather than modifying HTTP headers. The paper presents this combination as production-safe zero-code distributed tracing that avoids source-code modification and does not require kernel lockdown to be disabled.

3. Architecture and correlation mechanisms in the tracing system

The tracing-system architecture has two main components: eBPF Agents on each node and a centralized Trace Processor (Phan et al., 15 Aug 2025). The eBPF agents capture span events from system calls such as send, recv, read, and write; generate ingress and egress spans; perform intra-service and inter-service correlation locally; and forward correlated spans to the Trace Processor. The Trace Processor reconstructs complete traces, assigns a unique trace ID to related spans, and exports them to storage and visualization backends including Jaeger, Zipkin, and Grafana.

The data flow is explicitly staged. eBPF programs hook into socket and syscall entry and exit points to collect metadata, including socket tuples, PIDs, and timestamps. Kernel-space maps then match request and response events into spans. Cross-service correlation modules embed and extract span IDs in TCP option headers. Cross-thread correlation modules infer intra-service parent-child relationships via a lightweight greedy algorithm. Correlated spans, now annotated with parent span IDs, are sent to the Trace Processor for final trace-ID assignment.

Span generation is defined operationally. An ingress span runs from recv() to send() of a response, while an egress span runs from send() of a request to recv() of its response. Per-event metadata includes remote and local sockets, protocol, PID, timestamp, SpanID, and span type.

For intra-service correlation, the paper defines a delay feature vector for an ingress span that produces $n$ downstream calls: $$d_1 = \mathrm{start\_time}(\mathrm{egress}_1) - \mathrm{start\_time}(\mathrm{ingress}), \quad d_2 = \mathrm{start\_time}(\mathrm{egress}_2) - \mathrm{end\_time}(\mathrm{egress}_1), \quad \dots, \quad d_{n+1} = \mathrm{end\_time}(\mathrm{ingress}) - \mathrm{end\_time}(\mathrm{egress}_n).$$ Without pruning, the paper notes $O(m^n)$ combinations for $m$ egress spans and $n$ calls per request. It therefore applies an adaptive threshold

$T_k = \delta \cdot \mu_k$

with $\delta$ chosen empirically, for example $\delta = 4$, to cover approximately the 99th percentile of log-normal or exponential delays. After thresholding, the complexity per ingress is described as roughly $O(k \cdot \log m)$.

The four-step algorithm then proceeds as follows. First, it identifies potential candidates using the adaptive threshold. Second, it extracts a high-certainty set using the Central Deviation Score

$C(i) = \sum_k \frac{|d_k - \mu_k|}{\mu_k},$

and classifies an ingress as high-certainty if $$d_1 = \mathrm{start\_time}(\mathrm{egress}_1) - \mathrm{start\_time}(\mathrm{ingress}), \quad d_2 = \mathrm{start\_time}(\mathrm{egress}_2) - \mathrm{end\_time}(\mathrm{egress}_1), \quad \dots, \quad d_{n+1} = \mathrm{end\_time}(\mathrm{ingress}) - \mathrm{end\_time}(\mathrm{egress}_n).$$0, with $$d_1 = \mathrm{start\_time}(\mathrm{egress}_1) - \mathrm{start\_time}(\mathrm{ingress}), \quad d_2 = \mathrm{start\_time}(\mathrm{egress}_2) - \mathrm{end\_time}(\mathrm{egress}_1), \quad \dots, \quad d_{n+1} = \mathrm{end\_time}(\mathrm{ingress}) - \mathrm{end\_time}(\mathrm{egress}_n).$$1 given as an example. Third, it fits parametric PDFs—Normal, Log-Normal, or Exponential—using KS and Anderson-Darling tests and BIC, and falls back to a Gaussian Mixture Model if none fits well. Candidate combinations are scored by a total log-likelihood,

$$d_1 = \mathrm{start\_time}(\mathrm{egress}_1) - \mathrm{start\_time}(\mathrm{ingress}), \quad d_2 = \mathrm{start\_time}(\mathrm{egress}_2) - \mathrm{end\_time}(\mathrm{egress}_1), \quad \dots, \quad d_{n+1} = \mathrm{end\_time}(\mathrm{ingress}) - \mathrm{end\_time}(\mathrm{egress}_n).$$2

Fourth, ingress spans are sorted by their PDS gap, then greedily assigned; conflicts caused by reusing an egress span are resolved through a small local brute-force optimization that maximizes the summed PDS over the conflicting spans.

For inter-service correlation, the egress-side syscall-layer eBPF program reserves a map entry keyed by socket tuple and writes the SpanID; a socket-layer eBPF program uses bpf_reserve_hdr_opt and bpf_store_hdr_opt to embed the SpanID into a TCP option; the server-side socket-layer program uses bpf_load_hdr_opt to read the option and write the SpanID into a map; and the syscall exit program retrieves it to correlate the newly created ingress span. The paper states that this uses only eBPF helper functions available since Linux 5.10, applies a socket flag per traced microservice so that only selected TCP connections carry the extra option, and maintains a threat model with unmodified application-layer protocols and no exposure of trace IDs to untrusted processes.

4. Empirical results, overheads, and limitations of the tracing system

The evaluation uses a single physical server with an Intel i7-8700K, 64 GB RAM, Ubuntu 22.04, and kernel 6.5, running the DeathStarBench Hotel Reservation application with six services under Docker Compose (Phan et al., 15 Aug 2025). Concurrency levels are 250, 500, 750, 1000, 1250, and 1500 parallel requests per container. TraceWeaver, in a Python implementation, is the reported baseline.

For cross-thread correlation accuracy, CrossTrace achieves 98% accuracy for both Frontend and Search services at 250 to 500 concurrency. At 1000 to 1500 concurrency it achieves at least 90% accuracy for Frontend and approximately 88% for Search. TraceWeaver is reported as similar at low load but declining more rapidly beyond 1000, while a timing-only naïve closest-span approach drops below 75% at high concurrency.

For runtime, the adaptive threshold $$d_1 = \mathrm{start\_time}(\mathrm{egress}_1) - \mathrm{start\_time}(\mathrm{ingress}), \quad d_2 = \mathrm{start\_time}(\mathrm{egress}_2) - \mathrm{end\_time}(\mathrm{egress}_1), \quad \dots, \quad d_{n+1} = \mathrm{end\_time}(\mathrm{ingress}) - \mathrm{end\_time}(\mathrm{egress}_n).$$3 reduces candidate-finding runtime by 2–4× relative to fixed thresholds such as $$d_1 = \mathrm{start\_time}(\mathrm{egress}_1) - \mathrm{start\_time}(\mathrm{ingress}), \quad d_2 = \mathrm{start\_time}(\mathrm{egress}_2) - \mathrm{end\_time}(\mathrm{egress}_1), \quad \dots, \quad d_{n+1} = \mathrm{end\_time}(\mathrm{ingress}) - \mathrm{end\_time}(\mathrm{egress}_n).$$4 ms. Total correlation solving remains below 2 s at all evaluated loads for CrossTrace, whereas TraceWeaver grows from approximately 5 s at concurrency 250 to more than 300 s at 1500, corresponding to a 20–200× slowdown relative to CrossTrace.

For inter-service propagation overhead, the measured helper-function costs over 50,000 requests are less than 50 ns for bpf_reserve_hdr_opt, less than 300 ns for bpf_store_hdr_opt, and less than 600 ns for bpf_load_hdr_opt. The end-to-end latency impact is reported as less than 1 $$d_1 = \mathrm{start\_time}(\mathrm{egress}_1) - \mathrm{start\_time}(\mathrm{ingress}), \quad d_2 = \mathrm{start\_time}(\mathrm{egress}_2) - \mathrm{end\_time}(\mathrm{egress}_1), \quad \dots, \quad d_{n+1} = \mathrm{end\_time}(\mathrm{ingress}) - \mathrm{end\_time}(\mathrm{egress}_n).$$5s per RPC added, dominated by kernel traversal and much less than 1% of typical RPC latencies. CPU utilization is less than 5% of one core when tracing 1,000 req/s.

The paper also details implementation constraints. Kernel hooks use syscall entry and exit via kprobes and eBPF ring buffers, plus socket hooks at the TCP layer. Shared data structures use eBPF hash maps for in-flight spans and span-ID propagation. The user-space agent is written in Go and uses libbpf/goebpf to load BPF programs and collect completed spans via a perf ring. The stated dependencies are Linux 5.10+, tested on 6.5 with Ubuntu 22.04, LLVM/Clang at least 13, the BPF toolchain, Go at least 1.20, and OpenTelemetry Collector compatible output.

The paper identifies several limitations. Nearly identical egress-ingress delays under high concurrency can create collisions and cause approximately 10% of spans to be mis-correlated. As a mitigation, CrossTrace can return top-$$d_1 = \mathrm{start\_time}(\mathrm{egress}_1) - \mathrm{start\_time}(\mathrm{ingress}), \quad d_2 = \mathrm{start\_time}(\mathrm{egress}_2) - \mathrm{end\_time}(\mathrm{egress}_1), \quad \dots, \quad d_{n+1} = \mathrm{end\_time}(\mathrm{ingress}) - \mathrm{end\_time}(\mathrm{egress}_n).$$6 candidates for suspicious requests such as HTTP 502s or long-tail latencies, which increases the include-correct rate to more than 95% at the cost of trace-size overhead. Encrypted transport, including TLS, currently prevents TCP-option inspection; the paper proposes future work based on TLS record-layer hooks or additional sidecar support. It also states explicit trade-offs: larger $$d_1 = \mathrm{start\_time}(\mathrm{egress}_1) - \mathrm{start\_time}(\mathrm{ingress}), \quad d_2 = \mathrm{start\_time}(\mathrm{egress}_2) - \mathrm{end\_time}(\mathrm{egress}_1), \quad \dots, \quad d_{n+1} = \mathrm{end\_time}(\mathrm{ingress}) - \mathrm{end\_time}(\mathrm{egress}_n).$$7 reduces false negatives but enlarges the candidate space, and avoiding header modification preserves security but limits metadata payload to the TCP option budget of about 40 bytes.

5. CrossTrace as a cross-domain dataset of grounded scientific reasoning traces

In the scientific-reasoning literature, CrossTrace is a dataset rather than an observability system. It comprises 1,389 grounded reasoning traces drawn from three subsets: 518 biomedical traces from medRxiv and bioRxiv, 605 AI/ML traces from arXiv CS, and 266 cross-domain traces involving computational methods applied to biology or medicine (Bouras et al., 30 Mar 2026). After quality filtering with extraction confidence at least 0.6, the records are split by domain-stratified random sampling with seed 42 into 1,180 training records, 102 validation records, and 107 test records. The average length is approximately 605 tokens per record, and the extraction model is Claude Sonnet 4 in deterministic mode.

The dataset extends HypoGen’s Bit-Flip-Spark schema by inserting a source-grounded Trace between the input prior state and the output novel contribution. Formally,

$$d_1 = \mathrm{start\_time}(\mathrm{egress}_1) - \mathrm{start\_time}(\mathrm{ingress}), \quad d_2 = \mathrm{start\_time}(\mathrm{egress}_2) - \mathrm{end\_time}(\mathrm{egress}_1), \quad \dots, \quad d_{n+1} = \mathrm{end\_time}(\mathrm{ingress}) - \mathrm{end\_time}(\mathrm{egress}_n).$$8

with

$$d_1 = \mathrm{start\_time}(\mathrm{egress}_1) - \mathrm{start\_time}(\mathrm{ingress}), \quad d_2 = \mathrm{start\_time}(\mathrm{egress}_2) - \mathrm{end\_time}(\mathrm{egress}_1), \quad \dots, \quad d_{n+1} = \mathrm{end\_time}(\mathrm{ingress}) - \mathrm{end\_time}(\mathrm{egress}_n).$$9

$O(m^n)$0

and

$O(m^n)$1

Each reasoning step is 15–40 words and is paired with a direct quotation from the source paper. The paper explicitly contrasts this with a single narrative Spark paragraph: CrossTrace instead contains 3–6 discrete steps, each verifiable against paper text.

The discovery-pattern taxonomy assigns each record one primary pattern, plus any secondary patterns. The eight reported primary patterns are gap_fill with 820 records (59.0%), analogy_transfer with 256 (18.4%), mechanistic_link with 134 (9.6%), incremental_extension with 74 (5.3%), contradiction_exploit with 62 (4.5%), data_driven with 25 (1.8%), swanson_abc with 12 (0.9%), and replication with 5 (0.4%). The paper includes illustrative grounded traces for gap_fill, analogy_transfer, and other patterns, and emphasizes that each step is tied to a quotation and a section or paragraph reference.

A common misconception would be to treat these traces as ungrounded free-form rationales. The dataset description states the opposite: every step is intended to be source-grounded, paired with a quotation, and verifiable at step level.

6. Fine-tuning, evaluation, transfer, and validation for the dataset

The dataset paper fine-tunes Qwen2.5-7B-Instruct via QLoRA using the Axolotl framework (Bouras et al., 30 Mar 2026). The reported hyperparameters are LoRA rank $O(m^n)$2, $O(m^n)$3, application to all linear layers, 3 epochs, batch size 4 with gradient accumulation to simulate batch size 16, and learning rate $O(m^n)$4 with cosine decay. Training uses a single NVIDIA A100 on Google Colab Pro, with runtime of approximately 45 minutes per run. The procedure quantizes the base model to 4-bit, inserts low-rank adapters into all linear projection layers, freezes quantized weights, and trains only the LoRA adapters on CrossTrace chat-template data.

Evaluation uses IAScore from two LLM-as-judge systems, Structural Compliance, and cosine similarity for the hypothesis or spark. On the HypoGen test set of 50 records and the CrossTrace test set of 107 records, the untuned baseline versus Run 2b changes as follows: IAScore rises from 0.828 to 0.968 under GPT-4o judgment and from 0.716 to 0.888 under Claude Opus 4.5 judgment; hypothesis cosine similarity rises from 0.680 to 0.772; spark cosine similarity rises from 0.221 to 0.620; and Structural Compliance improves from 0% to 100%. The 95% confidence intervals reported for Run 2b are 0.946–0.986 for GPT-4o IAScore and 0.861–0.911 for Claude Opus 4.5 IAScore, obtained by bootstrap with 1,000 samples. The paper states that GPT-4o is consistently higher-calibrated than Claude, while both agree on relative improvements.

The domain-generalization analysis reports that balanced cross-domain training, described as Run 2b, outperforms single-domain CrossTrace-only fine-tuning, described as Run 1, on both CS-only HypoGen and multi-domain CrossTrace test sets despite a 50% CS-data dilution. In a controlled ablation with 1,180 records for every model, the mixed model using 590 CrossTrace and 590 HypoGen records achieves 99.3% of CrossTrace-only performance on the CrossTrace test and 99.7% of HypoGen-only performance on the CS test. The paper further notes that CrossTrace-only, which never saw CS alone, still scores 0.713 hypothesis cosine similarity on the HypoGen test. It interprets this as evidence that the form of reasoning, identified there as Trace structure, transfers across domains independently of domain-specific content.

Human validation is conducted on a stratified sample of 150 records comprising 678 steps. The reported results are 99.7% of steps grounded, 0.3% inferred, 0.0% fabricated, 59.0% grounding accurate, 41.0% grounding partial, 0.0% grounding incorrect, and 100% traces fully faithful. The paper states that no fabricated steps were identified and that 99.7% of steps were traceable to a genuine source quotation. It also states, explicitly as a qualified claim, that to its author’s knowledge CrossTrace is the first large-scale, cross-domain dataset with step-level grounded reasoning traces for hypothesis generation.

7. Conceptual relation between the two CrossTrace works

The two CrossTrace papers use the word trace in different technical senses. In the microservice paper, a trace is reconstructed from ingress and egress spans by correlating causal relationships across threads and services (Phan et al., 15 Aug 2025). In the dataset paper, a trace is an ordered reasoning chain of 3–6 discrete steps grounded in quotations from source papers (Bouras et al., 30 Mar 2026). The shared title therefore does not indicate shared method, codebase, or evaluation regime.

A plausible implication is that the term trace is being used as a general representation of structured dependency. In one setting, the dependency is operational and causal, connecting RPC spans into end-to-end execution traces. In the other, the dependency is epistemic and documentary, connecting prior knowledge, intermediate logical steps, and a novel hypothesis through step-level grounding. This suggests a superficial lexical commonality but not a common technical lineage.

The distinction matters for citation and retrieval. References to CrossTrace in systems or observability contexts point to the eBPF-based distributed tracing system. References to CrossTrace in scientific reasoning or hypothesis-generation contexts point to the cross-domain dataset and its QLoRA experiments. References to TRACE, despite the visual similarity, point instead to a deterministic, training-free hallucination-correction algorithm based on cross-layer candidate trajectories, and should not be conflated with either CrossTrace work (Ranade, 18 May 2026).