Active Task Disambiguation with LLMs (2502.04485v1)

Abstract: Despite the impressive performance of LLMs across various benchmarks, their ability to address ambiguously specified problems--frequent in real-world interactions--remains underexplored. To address this gap, we introduce a formal definition of task ambiguity and frame the problem of task disambiguation through the lens of Bayesian Experimental Design. By posing clarifying questions, LLM agents can acquire additional task specifications, progressively narrowing the space of viable solutions and reducing the risk of generating unsatisfactory outputs. Yet, generating effective clarifying questions requires LLM agents to engage in a form of meta-cognitive reasoning, an ability LLMs may presently lack. Our proposed approach of active task disambiguation enables LLM agents to generate targeted questions maximizing the information gain. Effectively, this approach shifts the load from implicit to explicit reasoning about the space of viable solutions. Empirical results demonstrate that this form of question selection leads to more effective task disambiguation in comparison to approaches relying on reasoning solely within the space of questions.

• The paper introduces a novel Bayesian method for Large Language Models to actively ask clarifying questions, efficiently reducing ambiguity in task specifications.
• Experiments demonstrate that explicitly selecting questions based on information gain significantly outperforms zero-shot methods, leading to more accurate task resolution.
• This framework improves LLM reliability and adaptability for real-world applications like interactive tutorials, intelligent coding assistance, and personalized education.

Active Task Disambiguation with LLMs

The paper "Active Task Disambiguation with LLMs" addresses a crucial aspect of LLMs: their capability to handle ambiguously specified tasks. While LLMs exhibit remarkable prowess across various domains such as logical reasoning, mathematical problem-solving, and creative writing, they encounter challenges when dealing with tasks that are not explicitly defined. This is particularly relevant in real-world applications where problem specifications might be inherently ambiguous or deliberately underspecified.

Core Contributions and Methodology

The authors propose a novel approach, grounded in the principles of Bayesian Experimental Design (BED), to mitigate task ambiguity. Specifically, the research formalizes task ambiguity and posits clarifying questions as a means to refine the problem space. By leveraging a BED framework, the authors enable LLMs to identify and ask questions that optimize informational gain. This emphasis on active question generation helps shift the focus from implicit reasoning to explicit consideration of viable solutions.

The methodology revolves around generating and selecting questions that can best partition the solution space efficiently, thus enhancing the process of active task disambiguation. Importantly, the paper introduces a utility function to evaluate the potential information gain of a given question. This utility function considers not only the informational gain but also the cost associated with obtaining clarification.

Experimental Design

The experiments are structured to validate two critical hypotheses: (1) LLMs, out of the box, may struggle with effective question generation due to limited pre-training exposure to high-quality clarifying questions, and (2) shifting the question selection process from implicit to an explicitly sampled space of solutions can improve performance.

Two experimental setups showcase these hypotheses:

1. 20 Questions Game: The classic setting is used to simulate a game where LLMs attempt to guess an entity by asking a series of yes-or-no questions. The experiments demonstrate that explicitly sampled solutions lead to better question generation and subsequently higher task accuracy.
2. Code Generation Task: Here, the application is expanded to generating code based on ambiguous user requirements. This setup uses input-output pairs to further specify tasks and resolves clarifications by executing test cases, thus ensuring minimal evaluation noise.

Results and Implications

The results indicate that approaches based on explicit information gain (EIG) consistently outperform traditional methods. Notably, questions derived through EIG produce more balanced partitions of solution spaces, ultimately leading to more accurate and efficient task resolutions compared to zero-shot question generation.

This framework holds significant implications for both theoretical exploration and practical applications in AI. By improving LLMs' ability to deal with ambiguity, this research enhances the reliability and adaptability of AI systems in dynamic and real-world settings. Practically, these advancements can lead to more nuanced interactions in domains like interactive tutorials, intelligent coding assistance, and personalized education systems.

Speculation on Future Developments

Future research could explore domain-specific applications and optimize the proposed methodologies with task-specific nuances. For instance, integrating more sophisticated models of user preference within the BED framework could refine the granularity and perceived relevance of questions asked by the LLM. Moreover, fine-tuning LLMs with datasets specifically tailored for clarifying question generation could bridge the observed gap between the out-of-the-box and optimized questions, effectively enhancing generalization and adaptability across diverse tasks.

In conclusion, this paper significantly contributes to understanding and enhancing LLMs' capabilities in interactive contexts, ensuring that AI not only comprehends but successfully navigates the complexities of natural language and human intention.