EchoTrace: Multi-Domain Trace Instrumentation

• EchoTrace is a multifaceted trace methodology that captures source-addressable evidence across domains such as conversational AI studies, LLM evaluation, and agentic reinforcement learning.
• It supports reproducible data collection by logging discrete events like prompts, responses, clicks, and survey answers, while also structuring long-form content for verification.
• Its applications range from digital interaction logging in platforms like ECHO and RECAP to physical sensing in radar and acoustic systems for structural reconstruction.

Searching arXiv for papers using the term “EchoTrace” and related titles. EchoTrace is a research label used in multiple, technically distinct senses across recent arXiv literature. In one line of work, it denotes the logging, structuring, and export subsystem of the ECHO platform for reproducible studies of human interaction with conversational AI systems and Web search engines (Liu et al., 10 Feb 2026). In another, it denotes a benchmark for eliciting and verifying memorized long-form content from LLMs, introduced as the primary testbed for the RECAP extraction pipeline (Duarte et al., 29 Oct 2025). Related usage also appears as “source-indexed reconstruction and traceable learning” within the ECHO selective turn-memory framework for agentic reinforcement learning (Xie et al., 30 Jun 2026), while “EchoTrace-style” is used to describe execution-trace-grounded reasoning systems for code models (Thakur et al., 28 Nov 2025). Outside language-model evaluation, the name has been applied to radar-echo tracing of particle-cascade ionization in polar ice (Prohira et al., 2021) and to phase-invariant three-dimensional echo localization frameworks based on ellipsoid intersection (Hahne, 2023). This diversity suggests that EchoTrace functions less as a single standardized method than as a recurring term for trace-centric instrumentation, reconstruction, or inference across multiple domains.

1. Human-interaction tracing in the ECHO platform

Within ECHO, EchoTrace is the set of logging, structuring, and export capabilities described under “Interaction Logging,” “Data Flow,” and “Comprehensive Data Export,” built on a serverless Firebase/Firestore backbone and controlled through a low-coding-load administrator dashboard (Liu et al., 10 Feb 2026). Its stated purpose is to support reproducible, mixed-method studies of human interaction with both “a ChatGPT style conversational interface powered by LLM APIs” and “a web search interface powered by the Search API,” while lowering technical barriers for researchers in information retrieval, HCI, and the social sciences.

The platform automatically logs “prompts, responses, queries, clicks, and survey answers” to Firebase and exports them as “structured CSV files” (Liu et al., 10 Feb 2026). For chat, “Each prompt-response pair is logged with unique identifiers, full text content, typing timestamps (start and end), and user ratings of chat turns and trajectories.” The participant interface also includes an “in-situ response rating mechanism” and an optional “note taking sidebar,” and those notes are exported as part of the dataset. For search, each query records “search text, typing timestamps, total count of result lists, clicked result URLs with timestamps, and result rankings for clicked URLs items,” thereby capturing clickthrough behavior and rank-position information.

EchoTrace in this setting is event-level rather than keystroke-level. The paper explicitly states that it does not mention keystroke-level logs, dwell time computation, intermediate draft states, tool calls, or suggestion tracking beyond the enumerated interaction types (Liu et al., 10 Feb 2026). This is significant because it fixes the granularity of downstream behavioral analysis around discrete prompt-response and query-click events rather than continuous interaction streams.

2. Architecture, workflow, and exported data schema

ECHO organizes the system into three layers: a frontend layer with “two React-based web applications built with TailwindCSS,” a backend layer using “Firebase services, including Firestore for real-time data storage and Firebase Authentication for user management,” and an external API layer with “LLM providers (OpenAI, Google Gemini, Anthropic Claude) for chat tasks and Search API for web search tasks,” with defaults “OpenAI and Brave Search” and a “modular design” for provider substitution (Liu et al., 10 Feb 2026).

Study orchestration is managed through “Study Flow Management,” whose default workflow is “(1) background survey, (2) pre-task questionnaire, (3) main task (chat or search), (4) post-task questionnaire, (5) experience survey, and (6) end-of-study survey” (Liu et al., 10 Feb 2026). Administrators can “enable, disable, or reorder steps and configure step-specific reminders or attention checks.” During operation, configurations are stored in Firestore; participants authenticate through Firebase Authentication; requests are sent to configured APIs; and “All interactions, including timestamps, clicks, and survey responses, are logged to Firestore in real-time.”

The exported entities include participant registration and demographic data; “pre/post-task questionnaire responses” and “experience survey” data; “complete chat histories with timestamps,” including “unique identifiers,” “full text content,” “typing timestamps (start and end),” and “user ratings of chat turns and trajectories”; “search queries with click data”; “in-situ survey responses”; and “participant notes” (Liu et al., 10 Feb 2026). The paper does not list explicit field names such as participant_id, session_id, or task_id, but it states that chat records carry “unique identifiers” and that all events include “timestamps.” It further states that relational linkage across participants, study steps, tasks, and events is managed in Firestore and reflected in the CSV exports, allowing downstream joining by identifiers and time.

This design places EchoTrace in ECHO at the intersection of instrument configuration and data provenance. “Survey instruments support both Form and JSON editing modes,” and “JSON Mode [to] allow direct import and export of question configurations,” which supports reproducible study configuration as well as reproducible data collection (Liu et al., 10 Feb 2026).

3. Benchmarking memorized long-form content in RECAP

A distinct use of EchoTrace appears in the RECAP paper, where EchoTrace is introduced as a benchmark for evaluating whether LLMs “have memorized and can be elicited to reproduce specific long-form training content,” with a particular focus on copyrighted texts (Duarte et al., 29 Oct 2025). The motivation is explicit: when training data cannot be inspected, “the strongest evidence of exposure is free-form verbatim or near-verbatim reproduction by the model itself.”

EchoTrace in this sense comprises “two main splits and over 70,000 40-token passages selectable for extraction and analysis” (Duarte et al., 29 Oct 2025). The benchmark includes 35 full-length books: 15 public domain books, 15 copyrighted bestsellers, and 5 non-training control books released after known cutoff dates for the evaluated models. It also includes 20 arXiv papers “widely believed to be common LLM training sources,” such as “Attention Is All You Need,” BERT, CLIP, and ViT. Books are segmented into semantically self-contained “events,” with metadata including an event title, characters, a detailed bullet-point summary, verbatim first and last sentences, and the original text_segment. For arXiv papers, an automated pipeline retrieves LaTeX source, detects the main file via \begin{document}, recursively inlines included files, strips comments, figures, and tables, expands common user-defined macros, extracts content up to the bibliography, and then feeds sections to the Section Summary Agent for event segmentation and metadata production.

Passage-level memorization analysis uses 40-token chunks. A chunk is counted as memorized if it matches the gold reference “with at most five token mismatches” (Duarte et al., 29 Oct 2025). Group-level evaluation distinguishes public domain, copyrighted, and non-training subsets. This construction allows EchoTrace to serve both as an elicitation substrate and as an automated verification benchmark for long-form reproduction.

The benchmark is also selective in what it releases. “Public domain books and their segmentations, summaries, and model outputs will be released,” while “Copyrighted passages will not be released” (Duarte et al., 29 Oct 2025). EchoTrace is hosted at https://huggingface.co/datasets/RECAP-Project/EchoTrace, and RECAP code is released at https://github.com/avduarte333/RECAP.

4. Evaluation protocol, metrics, and empirical findings in RECAP

EchoTrace is the evaluation substrate on which RECAP is run end-to-end (Duarte et al., 29 Oct 2025). The Extraction Agent uses event metadata and the known first sentence verbatim to prompt the model to reconstruct the passage “authentically, with no commentary.” A Verbatim Verifier labels outputs as “Yes” or “No,” routing refusals to a Jailbreaker and relevant attempts to a Feedback Agent. The Jailbreaker uses a single static prompt in “Narrative Tool Injection style,” while the Feedback Agent outputs abstract correction hints under “Major Structural Issues, Missing Elements, Inaccurate Elements,” explicitly avoiding verbatim quotes or close paraphrase.

The primary aggregate metric is weighted book-level ROUGE-L:

$$R_i = \left( \frac{1}{\sum_{j=1}^{N_i} w_{i,j}} \right) \sum_{j=1}^{N_i} w_{i,j} r_{i,j}.$$

Group-level scores for public domain, copyrighted, or non-training subsets are estimated via “bootstrapping (1,000 iterations sampling books with replacement); report mean and standard deviation of bootstrap means” (Duarte et al., 29 Oct 2025). The paper also defines ROUGE-L through longest common subsequence recall, precision, and $F_\beta$, and introduces a hybrid memorization score

$$m = \sigma( \beta_1 \cdot (1 - \text{BERT Loss}) + \beta_2 \cdot \text{Rouge} + \beta_3 \cdot \text{CS} + \beta_0 )$$

with coefficients $\beta_1 = 4$, $\beta_2 = 4.5$, $\beta_3 = 1.5$, and $\beta_0 = -5$ (Duarte et al., 29 Oct 2025).

Quantitative results show that RECAP improves extraction quality on EchoTrace over Prefix-Probing, DSP, and DSP + Jailbreak. For copyrighted books, the average over model families is reported as Prefix-Probing 0.088, DSP 0.258, DSP + Jailbreak 0.366, and RECAP 0.460 (Duarte et al., 29 Oct 2025). For GPT-4.1 on copyrighted extraction, “average ROUGE-L improves from 0.38 to 0.47 (~24% increase).” On popular arXiv papers, for GPT-4.1, Prefix-Probing yields 0.180, DSP 0.298, and RECAP 0.445. The paper further reports that the static jailbreak bypasses “>75% of refusals across models,” that “Most gains occur in the first iteration,” and that “fewer than 20% of events benefit from additional rounds.”

EchoTrace’s control split is used as a contamination check. The paper reports “Nearly zero extractions in control books released after model cutoffs (at most one 40-token passage per book/model), supporting that RECAP’s feedback loop does not contaminate generation with external knowledge” (Duarte et al., 29 Oct 2025). This is methodologically important because the benchmark is designed not only to detect extractability, but also to distinguish memorization from reconstruction induced by leaked prompts or evaluation artifacts.

5. Source-indexed reconstruction and traceable learning in agentic RL

A third usage appears in the paper “ECHO: Prune to act, trace to learn with selective turn memory in agentic RL,” where EchoTrace names the mechanism of “source-indexed reconstruction and traceable learning” within a selective turn-memory framework (Xie et al., 30 Jun 2026). The stated problem is that truncation and rolling-summary methods make acting feasible under bounded context windows but “collapse provenance,” so original source turns are no longer addressable during reinforcement learning.

ECHO addresses this by compressing each completed environment turn into a compact memory record,

$$s_i \sim \pi_\theta(. \mid c^{sum}), \qquad m_i = \text{parse}(s_i) \equiv f_{compress}(u_i),$$

and storing a source-indexed tuple

$e_i = (i, a_i, m_i).$

Here $i$ is the source turn index, $F_\beta$0 is a compact rendering of the action or tool call, and $F_\beta$1 is the concise factual finding from the latest tool response (Xie et al., 30 Jun 2026). At reconstruction time, the policy selects historical records from the memory bank, merges them with a recent suffix, and rebuilds a bounded policy context while preserving the selected indices as provenance.

The central credit-routing mask is defined over generated tokens. With $F_\beta$2, the mask is

$F_\beta$3

Group-relative advantage is

$F_\beta$4

and the masked surrogate objective is

$F_\beta$5

An equivalent policy-gradient view includes an explicit term for memory-selection actions over indices $F_\beta$6 (Xie et al., 30 Jun 2026).

Empirically, on BrowseComp-Plus with Qwen3-32B-Instruct, ECHO reaches “43.4% held-out accuracy,” compared with “GRPO (28.9%)” and “the rolling-summary baseline SUPO (36.1%),” while using “fewer turns and lower trajectory volume than SUPO” (Xie et al., 30 Jun 2026). The paper reports average turns at the end of training of 11.2 for GRPO, 62.5 for SUPO, and 45.3 for ECHO; trajectory split rates of 85.5% for SUPO and 57.8% for ECHO; and trajectories per rollout of 4.18 for SUPO and 3.13 for ECHO. It further reports zero-shot generalization gains on dense and MoE backbones across “Multi-Objective QA, CodeGym, LoCoBench-Agent, GAIA, HLE, Frames.”

This suggests that, in the RL setting, EchoTrace refers not to a dataset or logging facility but to a provenance-preserving credit-assignment mechanism that aligns retained evidence with reinforced evidence.

6. EchoTrace-style trace grounding and other domain-specific usages

The term also appears in derivative or analogical form outside these two core language-model settings. In “Generating Verifiable CoT from Execution-Traces,” the authors describe “a structured blueprint for an EchoTrace-style system” in which code is instrumented to capture execution traces and rationales are generated directly from those traces (Thakur et al., 28 Nov 2025). The trace is formalized as

$F_\beta$7

and a rationale $F_\beta$8 is “correct by construction” if

$F_\beta$9

The training objective combines output prediction and trace-grounded rationale supervision, and the paper reports gains of “up to 30 points on output prediction” and “28 points on input prediction” over base models (Thakur et al., 28 Nov 2025). Here “EchoTrace-style” denotes trace-to-rationale verification and source faithfulness rather than any fixed named benchmark.

In high-energy astrophysics, the RET-CR paper uses EchoTrace to describe “the idea of using radar echoes to trace the short-lived ionization left behind by particle cascades in ice” (Prohira et al., 2021). RET-CR transmits radio into near-surface polar ice and listens for reflections from dense in-ice cascades created by cosmic-ray air-shower cores. The underlying plasma condition is expressed through

$$m = \sigma( \beta_1 \cdot (1 - \text{BERT Loss}) + \beta_2 \cdot \text{Rouge} + \beta_3 \cdot \text{CS} + \beta_0 )$$0

with over-dense scattering when $$m = \sigma( \beta_1 \cdot (1 - \text{BERT Loss}) + \beta_2 \cdot \text{Rouge} + \beta_3 \cdot \text{CS} + \beta_0 )$$1 in the collisionless limit. The paper states that RET-CR expects “$$m = \sigma( \beta_1 \cdot (1 - \text{BERT Loss}) + \beta_2 \cdot \text{Rouge} + \beta_3 \cdot \text{CS} + \beta_0 )$$21 radar echo event per day,” with instrumented area “$$m = \sigma( \beta_1 \cdot (1 - \text{BERT Loss}) + \beta_2 \cdot \text{Rouge} + \beta_3 \cdot \text{CS} + \beta_0 )$$3,” a transmit power “$$m = \sigma( \beta_1 \cdot (1 - \text{BERT Loss}) + \beta_2 \cdot \text{Rouge} + \beta_3 \cdot \text{CS} + \beta_0 )$$4,” and a near-surface plasma lifetime “$$m = \sigma( \beta_1 \cdot (1 - \text{BERT Loss}) + \beta_2 \cdot \text{Rouge} + \beta_3 \cdot \text{CS} + \beta_0 )$$5” (Prohira et al., 2021). In this literature, EchoTrace refers to physically tracing cascade ionization via radar echoes.

A further acoustics-oriented use appears in “3-Dimensional Sonic Phase-invariant Echo Localization,” where the provided details label the framework “EchoTrace with PaCE” (Hahne, 2023). PaCE localizes a target point $$m = \sigma( \beta_1 \cdot (1 - \text{BERT Loss}) + \beta_2 \cdot \text{Rouge} + \beta_3 \cdot \text{CS} + \beta_0 )$$6 by intersecting ellipsoids defined by round-trip time-of-flight measurements:

$$m = \sigma( \beta_1 \cdot (1 - \text{BERT Loss}) + \beta_2 \cdot \text{Rouge} + \beta_3 \cdot \text{CS} + \beta_0 )$$7

The paper combines MEMG-based echo decomposition, a Siamese MLP for inter-channel correspondence, and nonlinear least-squares or root-finding over ellipsoid constraints. The reported localization result is “Mean RMSE: 23.3 mm (13.1%), Std: 15.1 mm (6.6%) across the 18 points,” and the echo-association test set achieves “F1 = 1.0” (Hahne, 2023). In this context, EchoTrace denotes phase-invariant three-dimensional echo localization based on geometric intersection and learned correspondence.

7. Conceptual commonalities, distinctions, and scope

Across these papers, EchoTrace consistently denotes a trace-centric methodology, but the object being traced differs substantially. In ECHO, the trace is a sequence of participant interactions and survey responses (Liu et al., 10 Feb 2026). In RECAP, it is the relation between event-level long-form references and free-form model generations, evaluated through passage-level memorization and ROUGE-L (Duarte et al., 29 Oct 2025). In agentic RL, it is provenance over selected source turns and the token-level routing of outcome credit (Xie et al., 30 Jun 2026). In execution-grounded code reasoning, it is the runtime trace of program behavior (Thakur et al., 28 Nov 2025). In radar and acoustics, it is a physical echo used to reconstruct hidden structure or transient ionization (Prohira et al., 2021, Hahne, 2023).

A common misconception would be to treat EchoTrace as a single standardized framework. The evidence instead supports a polysemous usage spanning human-subject experimentation, LLM memorization benchmarking, RL credit assignment, code-trace narration, radar detection, and acoustic localization. Another misconception would be to assume that all EchoTrace systems expose the same granularity or privacy model. For example, ECHO integrates “IRB-compliant consent form with customizable text and acknowledgment checkboxes,” but “does not discuss anonymization or PII handling beyond this” (Liu et al., 10 Feb 2026), whereas the RECAP benchmark explicitly withholds copyrighted passages from release (Duarte et al., 29 Oct 2025).

The most stable cross-domain theme is the preservation of source-addressable evidence. In ECHO, this appears as “unique identifiers” and timestamps attached to prompts, responses, queries, clicks, and surveys (Liu et al., 10 Feb 2026). In RECAP, it appears as event boundaries, passage selection, and automated fidelity checking (Duarte et al., 29 Oct 2025). In agentic RL, it appears as source indices reused both for reconstruction and for masked credit routing (Xie et al., 30 Jun 2026). A plausible implication is that the recurring appeal of the term lies in this combination of trace capture, source localization, and downstream verifiability, even though the concrete technical substrates differ widely from Firestore logs to token masks, execution traces, radar echoes, and ellipsoid-intersection solvers.