YIELD: A Large-Scale Dataset and Evaluation Framework for Information Elicitation Agents

Abstract: Most conversational agents (CAs) are designed to satisfy user needs through user-driven interactions. However, many real-world settings, such as academic interviewing, judicial proceedings, and journalistic investigations, involve broader institutional decision-making processes and require agents that can elicit information from users. In this paper, we introduce Information Elicitation Agents (IEAs) in which the agent's goal is to elicit information from users to support the agent's institutional or task-oriented objectives. To enable systematic research on this setting, we present YIELD, a 26M-token dataset of 2,281 ethically sourced, human-to-human dialogues. Moreover, we formalize information elicitation as a finite-horizon POMDP and propose novel metrics tailored to IEAs. Pilot experiments on multiple foundation LLMs show that training on YIELD improves their alignment with real elicitation behavior and findings are corroborated by human evaluation. We release YIELD under CC BY 4.0. The dataset, project code, evaluation tools, and fine-tuned model adapters are available at: https://github.com/infosenselab/yield.

• The paper introduces YIELD—a large-scale, ethically curated dataset that benchmarks information elicitation agents distinct from user-centric conversational models.
• It formalizes information elicitation as a finite-horizon POMDP and employs offline Advantage-Weighted Regression with LoRA adapters for effective policy optimization.
• The research highlights novel metrics like Conformity, Progression, and Turn-Length Ratio, underscoring practical and ethical considerations in institutional dialogue.

YIELD: A Dataset and Evaluation Framework for Information Elicitation Agents

Introduction and Motivation

The paper introduces YIELD, a large-scale, ethically curated dataset and benchmark designed specifically for the development and evaluation of Information Elicitation Agents (IEAs), a class of conversational agents optimized for extracting information to fulfill institutional or task-oriented objectives, rather than fulfilling user-initiated goals. Unlike traditional conversational agents (CAs), which prioritize satisfying user requests, IEAs are tailored for scenarios such as academic interviews, judicial proceedings, and journalistic investigations where strategic elicitation of information is essential for downstream decision-making processes.

Figure 1: Core distinctions between traditional conversational agents and information elicitation agents are highlighted across initiative, agent objectives, and use cases.

The authors formally distinguish IEAs from user-centric CAs along the dimensions of conversational initiative, primary agent objective, and expected dialogue dynamics. They argue that progress in this subfield is hindered by the lack of suitable datasets reflecting real elicitation behavior and establish YIELD as the first large-scale resource to address this gap.

YIELD Dataset Construction and Characteristics

YIELD comprises 2,281 dialogues, 390,205 annotated turns, and 26M tokens, sourced exclusively from public-domain and Creative Commons-licensed repositories. The dataset spans four domains bearing significant institutional context: Oral History, Judicial Proceedings, Academic Interviews, and Journalistic Investigations. Rigorous manual annotation harmonized heterogeneous transcript formats, standardized speaker labels, and ensured role-level markup critical for modeling elicitation dynamics. The dataset is stratified into training, development, and test partitions for robust experimental protocols.

The dialogues in YIELD are an order of magnitude longer (average 171 turns) than mainstream task-oriented datasets such as MultiWOZ, SGD, and DSTC2. This scale enables research on multi-turn coherence, context accumulation, and turn-wise strategic information seeking, aspects largely absent in short, user-driven dialogues.

Formalization of the Information Elicitation Problem

The authors model information elicitation as a finite-horizon Partially Observable Markov Decision Process (POMDP), where the agent (elicitor) interacts with an environment (respondent) holding latent knowledge states. At each turn, the agent observes only the dialogue context and must select utterances to maximize cumulative "informational reward," operationalized via a proxy for factual novelty. The information state is encoded using token representations from a causal LLM, mirroring the evolving belief state in POMDP theory.

The policy (elicitor utterance generator) is learned via offline Advantage-Weighted Regression (AWR), a variant of actor-critic RL that updates policy and value head parameters based on weighted empirical returns. Low-rank adaptation (LoRA) adapters allow parameter-efficient fine-tuning, fixing the base LLM.

Task-Specific Metrics for Elicitation

The paper introduces IEA-appropriate evaluation criteria going beyond conventional metrics such as BLEU or task success:

• Conformity: Distributional match to real elicitation (perplexity, response length).
• Progression: Measures topic advancement and penalizes stagnation, computed as average decayed cosine distance between sequential utterance embeddings.
• Turn-Length Ratio (TLR): Quantifies the expected asymmetry of elicitation, where effective IEAs engineer significantly longer respondent than elicitor turns.

These diagnostic metrics are implemented in an open-source Python toolkit, supporting reproducibility.

Experimental Results

Pilot experiments employ Llama-3.1-8B, DeepSeek-R1-Distill-Llama-8B, and Llama-3.2-3B-Instruct as backbone LLMs with PEFT and LoRA adapters. Policies are trained under both supervised fine-tuning (SFT) and offline RL (ORL, AWR approach), and contrasted with prompt-only (zero-shot) variants.

Findings:

• Conformity: Fine-tuned agents (SFT/ORL) achieve human-analogous perplexity and response lengths; prompt-only models exhibit substantial stylistic and length errors, often failing to align with human elicitor behaviors even with elaborate prompts. Prompted model outputs are notably verbose and less cost-effective in token usage.
• Progression: SFT/ORL models closely match human dialogue advancement, whereas prompt-only models are more repetitive and less likely to drive new information.
• Turn-Length Ratio: Realistic respondent/elicitor asymmetry is captured only by SFT/ORL; prompt-only models either reverse or neutralize this asymmetry.
• Human Evaluation: Crowd-sourced ratings suggest that while prompted responses are judged more favorably in absolute terms (likely due to verbosity and polish), SFT/ORL responses more faithfully reflect real elicitor behavior distributions.

Ethical and Practical Implications

The formalization and publication of YIELD enables reproducible research on agents that drive conversations for institutional objectives. However, the use of IEAs raises salient ethical risks around leading questions, power imbalances, data privacy, and potential bias, especially in sensitive domains. The authors explicitly recommend technical and procedural mitigations—such as restricted deployment, rigorous oversight, and post-generation filtering—to align IEA usage with prevailing ethical norms.

Limitations and Future Directions

The reward function in the offline RL protocol is intentionally minimal and domain-agnostic, using entity-level novelty, which may not perfectly capture all axes of "information value." Current evaluation is exclusively next-turn offline rather than interactive with live respondents, and certain metrics (such as Progression) are embedding-model dependent. The framework does not yet address multi-agent interaction, adaptive conversation management, or extended dialogue planning required in longer-horizon elicitation scenarios.

The work opens several promising directions:

• Enhanced reward shaping leveraging richer notions of informativeness, credibility, or subjective value.
• Iterative, interactive evaluation frameworks with human or simulated respondents.
• Extensions to institutional auditing, bias detection, and safe deployment mechanisms for IEAs.

Conclusion

YIELD provides a high-coverage, legally reusable foundation for research on information elicitation dialogue and the principled development and evaluation of IEA policies. The benchmark, together with the POMDP modeling approach, tailored metrics, and RL-based optimization, catalyzes systematic exploration of agent-driven information seeking, with applications in domains demanding nuanced and adaptive conversational strategies.

---