QA-TOOLBOX: Conversational Question-Answering for process task guidance in manufacturing (2412.02638v1)

Abstract: In this work we explore utilizing LLMs for data augmentation for manufacturing task guidance system. The dataset consists of representative samples of interactions with technicians working in an advanced manufacturing setting. The purpose of this work to explore the task, data augmentation for the supported tasks and evaluating the performance of the existing LLMs. We observe that that task is complex requiring understanding from procedure specification documents, actions and objects sequenced temporally. The dataset consists of 200,000+ question/answer pairs that refer to the spec document and are grounded in narrations and/or video demonstrations. We compared the performance of several popular open-sourced LLMs by developing a baseline using each LLM and then compared the responses in a reference-free setting using LLM-as-a-judge and compared the ratings with crowd-workers whilst validating the ratings with experts.

• The paper introduces QA-TOOLBOX that integrates LLM-augmented data for real-time, spec-grounded task guidance in manufacturing.
• The authors develop a robust dataset with over 200K QA pairs and a benchmarking tool to evaluate LLM performance against expert ratings.
• The evaluation framework demonstrates that higher parameter models outperform smaller ones in providing accurate, contextual responses to technicians.

Insights into QA-TOOLBOX: Conversational Question-Answering for Process Task Guidance in Manufacturing

The paper introduces QA-TOOLBOX, a system designed to enhance task guidance in manufacturing through conversational question-answering (QA), leveraging LLMs for data augmentation. The impetus to address challenges in manufacturing—such as complex procedure adherence and high attrition rates—places emphasis on developing AI-driven task guidance solutions. The authors present QA-TOOLBOX as both a dataset and a benchmarking tool, assessing the efficiency of LLMs to provide real-time, spec-grounded responses to technicians' queries.

Core Contributions

1. Dataset Composition and Augmentation: The authors compile a robust dataset from a blend of naturally occurring technician interactions and spec-driven task guidance requirements. This dataset includes over 200,000 QA pairs, encompassing narrations and video demonstrations. LLMs augment data by filling gaps, creating spec documents, and generating technician narrations akin to a real-world advanced manufacturing setting.
2. Benchmark Development: The dataset facilitates benchmarking by testing LLMs in scenarios requiring document-grounded QA. The task is non-trivial, necessitating the model's understanding of specification documents and sequenced actions. The paper's methodology involves comparing these LLM-generated answers against expert and crowd-worker ratings without ground-truth references, securing LLM-as-a-judge evaluations.
3. Evaluation Framework: The work discusses deploying various LLMs as judges—evaluating model responses based on correctness, conciseness, completeness, and groundedness—using a scalable, automated evaluation methodology. The comparison between several open-source LLMs, like Phi3 and Mistral, provides compelling insights into their performance and adaptation for manufacturing QA systems.

Findings and Results

Prominent findings reveal that higher parameter models like Phi-3-medium-128k offer superior performance across the core metrics, while models such as Llama3-8b-Instruct show promising results when balancing model size and effectiveness. The results emphasize the significance of model parameter scale, with larger models, albeit more processor-intensive, delivering more accurate and contextually relevant guidance.

An intriguing aspect is the correlation between LLMs and human ratings, with LLMs often aligning with or even surpassing human evaluations in certain respects. The stringent evaluation through configurations such as GPT4o, Nemotron, and Mixtral further confirms the robustness of LLM-as-judge when properly configured and supervised.

Theoretical and Practical Implications

Theoretically, this approach suggests new paradigms in evaluating AI's integrative capabilities in real-world task environments, extending current research into multimodal reasoning paradigms. Practically, QA-TOOLBOX proposes a scalable methodology to overcome high turnover rates and procedural complexities in manufacturing by offering personalized, immediate support to technicians through AI.

Future work may explore refining multimodal LLMs to incorporate long-term visual contexts to improve their task guidance capabilities due to the dynamic and non-deterministic nature of manufacturing procedures. Expansion into other domains with similar needs for real-time task guidance is also a viable exploration.

Overall, this paper delineates a compelling pathway for embedding AI into the manufacturing setting, proving its value as both a developmental catalyst for AI applications and a testbed for theoretical advancements in LLM capabilities.