DiscoTrace: Representing and Comparing Answering Strategies of Humans and LLMs in Information-Seeking Question Answering

Abstract: We introduce DiscoTrace, a method to identify the rhetorical strategies that answerers use when responding to information-seeking questions. DiscoTrace represents answers as a sequence of question-related discourse acts paired with interpretations of the original question, annotated on top of rhetorical structure theory parses. Applying DiscoTrace to answers from nine different human communities reveals that communities have diverse preferences for answer construction. In contrast, LLMs do not exhibit rhetorical diversity in their answers, even when prompted to mimic specific human community answering guidelines. LLMs also systematically opt for breadth, addressing interpretations of questions that human answerers choose not to address. Our findings can guide the development of pragmatic LLM answerers that consider a range of strategies informed by context in QA.

• The paper introduces DiscoTrace, a pipeline that formalizes answering strategies by encoding responses as discourse act–interpretation tuples using RST segmentation.
• The paper demonstrates, via cross-perplexity metrics on Reddit subcommunities, that human answer structures are diverse and context-specific while LLMs converge to a uniform breadth-first approach.
• The paper reveals that explicit prompting only marginally shifts LLM behavior, highlighting the need for strategy personalization to better align with human rhetorical complexity.

DiscoTrace: Structured Analysis and Comparison of Human and LLM Answer Strategies in Information-Seeking QA

Overview

"DiscoTrace: Representing and Comparing Answering Strategies of Humans and LLMs in Information-Seeking Question Answering" (2604.15140) introduces DiscoTrace, a pipeline for the formal representation and comparison of the answering strategies employed by humans and LLMs in information-seeking question answering (QA). DiscoTrace encodes each answer as a sequence of discourse act–interpretation tuples, mapped atop rhetorical structure theory (RST) segmentation, enabling rigorous dissection of both rhetorical structure and the specific interpretations of ambiguous or multifaceted questions that are actually addressed. The framework supports comparative analysis of diverse human communities, demonstrating that different subcommunities optimize distinct answering objectives. Conversely, LLMs lack comparable strategy diversity, converging toward uniform, breadth-preferring answering policies that deviate systematically from human conventions, even under explicit mimicking instructions.

DiscoTrace Formalism and Pipeline

DiscoTrace operates in three main stages. First, it segments answers using an RST parser, efficiently identifying action segments by coarsening elementary discourse units (EDUs) and using boundary relations that mark likely transitions between discourse acts.

Figure 1: RST trees yield discourse scaffolding; boundary nodes, whose children reside in distinct acts, define segment boundaries.

Each resulting segment receives a discourse act annotation from a 21-act ontology, grouped into five rhetorical action families (answering, commenting, information-seeking, redirecting, and no-op).

Figure 2: The action ontology—21 discourse acts, in five families—with explicit eligibility for pairing to question interpretations.

Finally, for interpretation-eligible acts, the pipeline pairs each segment with a specific question interpretation. Candidate interpretations are automatically generated with LLM prompting and deduplication, reflecting plausible readings rooted in explicit question ambiguity or community-anchored context. Annotator validation demonstrates strong agreement ($k \approx 0.7$ at the family level; 74% exact for interpretation pairing), confirming that DiscoTrace supports coherent, fine-grained structural comparisons.

Figure 3: Given a question and its interpretations, DiscoTrace scaffolds the answer into a sequence of discourse act/interpretation tuples.

Quantitative Analysis of Answering Strategies across Communities

Applying DiscoTrace to corpora from nine Reddit subcommunities, the analysis reveals marked rhetorical divergence across expert, lay, and experiential forums. Domain-agnostic QA forums (e.g., r/explainlikeimfive, r/NoStupidQuestions, r/OutOfTheLoop) display high intra- and inter-community cross-perplexity predictability in their answer structures, reflecting a small, generic rhetorical policy set. In contrast, expert-moderated communities (r/AskHistorians, r/AskEconomics, r/asklinguistics) exhibit low cross-predictability with the generic forums and differentiated rhetorical profiles, e.g., increased clarification and external source citation.

Figure 4: Train and eval on human answers demonstrates that expert/generic communities diverge in discourse act sequencing.

Shared topical focus between communities is an insufficient predictor of rhetorical alignment: for instance, r/history and r/AskHistorians both discuss history, but their act sequence distributions are notably distinct due to self-moderation and answer expectations. This indicates that rhetorical strategy, not just information content or forum topic, is a community-level phenomenon.

Figure 5: The proportion of answers addressing each of the 21 acts in r/explainlikeimfive and r/AskHistorians, with significant differences marked.

LLM Versus Human Rhetorical Behavior

LLMs, across both proprietary and open-source systems, default to a narrow spectrum of answering strategies, best approximating generic QA rather than specialized community norms. Cross-perplexity matrices for LLM answers trained on one community and evaluated on humans from another community show little strategy specialization, even when LLMs are instructed to mimic detailed community guidelines.

Figure 6: LLMs (here, claude-haiku-4.5) fail to reflect human answer strategy diversity—cross-perplexity on discourse act bigram models is high versus experts.

Prompting with specific community guidelines (e.g., r/AskHistorians' standards for sourcing and speculation, r/ScienceBasedParenting's evidence requirements) elicits marginal reductions in the gap but does not suffice to match human rhetorical complexity or breadth.

Figure 7: Prompting LLMs (claude-sonnet-4.5) with explicit guidelines fails to recover human-like answer strategies as measured by cross-perplexity.

Analysis of LLM-to-LLM act sequence cross-predictability demonstrates near invariance across question type or domain.

Figure 8: LLM outputs (qwen3-32b) exhibit extremely low perplexity when trained and evaluated on different communities—showcasing rigidity in strategy.

Selection and Depth of Addressed Interpretations

Humans select and focus on a subset of plausible question interpretations, driven by community norms, question ambiguity, and conversational pragmatics. Expert forums cover more interpretations than lay forums (e.g., 41% vs. 34% of generated interpretations addressed on average). Human answers also dedicate a larger proportion of eligible discourse segments to each selected interpretation, reflecting rhetorical depth and specificity.

Conversely, LLMs adopt a breadth-first policy, systematically "overanswering" by addressing low-probability or irrelevant interpretations rarely covered by humans—even more so when prompted to mimic expert communities. This overgeneration is quantifiable as LLMs rely on exhaustivity heuristics rather than pragmatic selection optimized for the user's inferred intent.

Figure 9: LLMs systematically overaddress, covering interpretations that humans rarely select; this tendency may worsen under mimic instructions.

Empirically, LLM answers are longer, less focused, and exhibit lower dedication values—multiple eligible segments dilute the rhetorical focus per interpretation—supporting the claim that LLMs currently lack the pragmatic discrimination present in human answers.

Theoretical and Practical Implications

DiscoTrace's representational rigor enables interpretable, compositional study of high-level answering strategy, separating rhetorical structure from simple factuality, style, or surface coherence. The empirical findings underscore the following:

• Supervised alignment or RLHF pipelines that collapse answerer diversity may systematically ignore important human preferences and answer structures, impairing LLM reliability in specialized or high-context QA.
• Strategy personalization for LLMs requires explicit modeling of both rhetorical acts and interpretation selection, using insights from community-level human behavior.
• DiscoTrace introduces a mechanism for plan-first answer generation, where a strategy (act/interpretation scaffold) is induced prior to content realization, enabling controllable, user-aligned answer structure.

Extending this paradigm beyond Reddit to platforms like StackOverflow or Quora, and into multi-turn, dynamic dialog, can enrich study of domain adaptation, context-aware answer planning, and robust user modeling.

Conclusion

DiscoTrace formalizes and elucidates core differences in information-seeking QA answer construction between human communities and LLMs. It demonstrates that LLMs neither reflect nor approximate the rhetorical nor interpretative diversity found among humans, even when provided explicit instructions. Future advances in LLM alignment and QA pragmatics will require integration of discourse-level objectives, context-aware rhetorical planning, and user-centric interpretation selection. DiscoTrace provides a reproducible, interpretable foundation for such developments.