QueryBuilder: Human-in-the-Loop Query Development for Information Retrieval (2409.04667v2)

Abstract: Frequently, users of an Information Retrieval (IR) system start with an overarching information need (a.k.a., an analytic task) and proceed to define finer-grained queries covering various important aspects (i.e., sub-topics) of that analytic task. We present a novel, interactive system called $\textit{QueryBuilder}$, which allows a novice, English-speaking user to create queries with a small amount of effort, through efficient exploration of an English development corpus in order to rapidly develop cross-lingual information retrieval queries corresponding to the user's information needs. QueryBuilder performs near real-time retrieval of documents based on user-entered search terms; the user looks through the retrieved documents and marks sentences as relevant to the information needed. The marked sentences are used by the system as additional information in query formation and refinement: query terms (and, optionally, event features, which capture event $'triggers'$ (indicator terms) and agent/patient roles) are appropriately weighted, and a neural-based system, which better captures textual meaning, retrieves other relevant content. The process of retrieval and marking is repeated as many times as desired, giving rise to increasingly refined queries in each iteration. The final product is a fine-grained query used in Cross-Lingual Information Retrieval (CLIR). Our experiments using analytic tasks and requests from the IARPA BETTER IR datasets show that with a small amount of effort (at most 10 minutes per sub-topic), novice users can form $\textit{useful}$ fine-grained queries including in languages they don't understand. QueryBuilder also provides beneficial capabilities to the traditional corpus exploration and query formation process. A demonstration video is released at https://vimeo.com/734795835

• The paper demonstrates an interactive system that enables novice users to iteratively refine cross-lingual queries with neural and probabilistic IR models.
• It employs a two-step workflow combining initial query creation with probabilistic retrieval and neural enrichment using pre-trained language models to boost relevance.
• The approach significantly improves retrieval performance and reduces query formulation time, achieving up to an 18% nDCG improvement compared to traditional methods.

Human-in-the-Loop Query Development for Information Retrieval: An Analysis of QueryBuilder

The paper presents "QueryBuilder," an interactive system designed for novice users to create fine-grained queries for Cross-Lingual Information Retrieval (CLIR) systems. This approach leverages a user-friendly interface and efficient IR mechanisms to refine and develop complex queries over iterative interactions. The system aims to cater to users who start with an overarching information need and gradually develop more specific sub-topics, streamlining the traditionally labor-intensive process of query formation.

System Architecture and Workflow

QueryBuilder facilitates the query formation process through an intuitive two-step workflow:

1. Initial Query Creation:
   • The user inputs initial search terms that define the broad information need.
   • The system uses a probabilistic IR model to retrieve relevant sentences from an English corpus.
   • Users then mark sentences as relevant, contributing to an evolving, refined query.
2. Query Enrichment:
   • Utilizes a Siamese network-based neural IR model to find sentences similar to those marked as relevant in the first step.
   • This neural IR process captures semantic nuances missed by solely lexical systems, improving the query’s effectiveness.
   • Users can refine the query iteratively by selecting further relevant sentences.

The probabilistic IR model operates based on term frequencies and weights, adapting dynamically with user interactions. In contrast, the neural IR model employs pre-trained BERT or XLM-R architectures to understand the high-level semantics of the user's query, thus ensuring a comprehensive retrieval process.

Experimental Evaluation

The efficacy of QueryBuilder was tested with Arabic-English CLIR tasks using the IARPA BETTER IR datasets. The experiments involved novice users who applied the QueryBuilder system to generate queries iteratively. Analyzed over eight overarching tasks with 54 sub-topics, the results showcased significant improvements in retrieval performance. Using Normalized Discounted Cumulative Gain (nDCG) as a metric, the performance of user-generated queries was notably close to that of queries crafted by experienced annotators at NIST.

Key results include:

• The nDCG improved markedly by 6-18% when user-selected sentences were added to search terms.
• Queries refined through the neural enrichment process yielded a further 1% improvement in nDCG.
• Overall, novice user queries formed through QueryBuilder outperformed basic overarching task queries, achieving up to 12% better results.

Comparative Analysis with NIST Workflow

A comparison with the existing NIST query development process highlighted several areas where QueryBuilder presented enhancements. The traditional NIST approach involves extensive manual search iterations and refinement, often demanding upwards of an hour per analytic task. QueryBuilder, however, significantly reduces this effort, enabling query formation in under 10 minutes through guided, iterative interactions.

Broader Implications and Future Directions

QueryBuilder's design aligns with the goals of making advanced IR systems accessible to non-expert users, thereby democratizing the power of CLIR systems. The implications are vast, extending to fields requiring rapid and nuanced information retrieval across languages, such as international research, intelligence analysis, and multilingual content curation.

Practical applications of QueryBuilder could lead to more responsive and adaptive search systems, allowing users to engage in cross-language searches without specialized knowledge of the algorithms backing their queries. The integration of probabilistic and neural IR methods reveals a promising direction for hybrid systems that can offer balance in speed and contextual understanding.

Future developments could focus on enhancing the real-time feedback mechanisms and exploring further refinements in neural IR capabilities. Additionally, scaling the system to handle larger and more diverse datasets would be a logical step, as would integrating multilingual support more deeply into the overall IR architecture.

In summary, QueryBuilder represents a significant step towards refining human-in-the-loop query development processes, enhancing both the efficiency and effectiveness of IR systems for a broad range of users. Its contributions to rapid query formulation and iterative enhancement demonstrate practical advancements in making sophisticated IR tools accessible and usable for novices and experts alike.