Do LLMs suffer from Multi-Party Hangover? A Diagnostic Approach to Addressee Recognition and Response Selection in Conversations (2409.18602v1)

Abstract: Assessing the performance of systems to classify Multi-Party Conversations (MPC) is challenging due to the interconnection between linguistic and structural characteristics of conversations. Conventional evaluation methods often overlook variances in model behavior across different levels of structural complexity on interaction graphs. In this work, we propose a methodological pipeline to investigate model performance across specific structural attributes of conversations. As a proof of concept we focus on Response Selection and Addressee Recognition tasks, to diagnose model weaknesses. To this end, we extract representative diagnostic subdatasets with a fixed number of users and a good structural variety from a large and open corpus of online MPCs. We further frame our work in terms of data minimization, avoiding the use of original usernames to preserve privacy, and propose alternatives to using original text messages. Results show that response selection relies more on the textual content of conversations, while addressee recognition requires capturing their structural dimension. Using an LLM in a zero-shot setting, we further highlight how sensitivity to prompt variations is task-dependent.

• The paper introduces a diagnostic pipeline that isolates structural features to assess LLM performance in multi-party conversations.
• It shows that detailed structural cues enhance addressee recognition while textual content is more critical for effective response selection.
• Results reveal that prompt verbosity and conversation complexity significantly affect LLM outcomes, guiding future improvements.

Investigating LLMs in Multi-Party Conversations: Addressee Recognition and Response Selection

In the paper "Do LLMs suffer from Multi-Party Hangover? A Diagnostic Approach to Addressee Recognition and Response Selection in Conversations," Nicolò Penzo and colleagues present a diagnostic and evaluative approach to assessing LLMs in the context of Multi-Party Conversations (MPCs). Notably, the authors aim to address the performance of LLMs in two fundamental tasks: Addressee Recognition (AR) and Response Selection (RS), with a particular focus on the influence of structural attributes of conversations.

Key Objectives and Approach

The paper's central objective is to develop a methodological pipeline for evaluating model performance on MPCs, leveraging a diagnostic approach that isolates specific structural features. This contrasts with conventional evaluation methods by emphasizing structural complexity rather than relying solely on traditional linguistic metrics.

Notably, the authors divide their diagnostic evaluation into several steps:

1. Modeling Input Representations: The authors consider four input representations for conversations, varying between pure textual and structural forms.
2. Classification Workflow: They implement a classification workflow involving Llama2-13b-chat, exploring different prompt verbosity levels and input combinations.
3. Diagnostic Datasets: The evaluation is conducted on subsets of the Ubuntu Internet Relay Chat corpus with fixed numbers of users, ensuring a controlled environment to analyze performance variations.

Experimental Details and Findings

Quantitative evaluations reveal distinct patterns in model performance based on the type of input and the inherent structural complexity of conversations:

1. Addressee Recognition (AR):
   • The interaction transcript (STRUCT) significantly enhances AR accuracy, underscoring the importance of structural data.
   • Summarizing conversations or describing users can mitigate data privacy concerns, yet their efficacy is suboptimal compared to detailed transcripts.
   • Prompt verbosity profoundly impacts AR performance, with verbose prompts generally yielding better results.
2. Response Selection (RS):
   • Textual content predominates in RS tasks, with conversation transcripts (CONV) delivering superior performance.
   • Unlike AR, RS performance is less sensitive to prompt verbosity, implying intrinsic alignment with the task's linguistic nature.
3. Structural Complexity:
   • Higher degree centrality degrades AR performance across all configurations, with models showing significant variance based on conversation complexity.
   • In RS, structural metrics reveal no clear correlation, suggesting the predominance of text over structural aspects in choosing appropriate responses.

Practical and Theoretical Implications

The findings hold both practical and theoretical significance. Practically, this paper contributes to developing more robust diagnostic tools for evaluating LLMs in MPCs. By isolating structural features and understanding their impact, future research can devise more sophisticated models that better integrate linguistic and structural cues.

Theoretically, the paper challenges the conventional wisdom of MPC modeling, opening avenues for hybrid approaches that balance both structural and linguistic aspects. Additionally, the authors highlight the potential of summarization and description techniques to enhance privacy and facilitate data sharing compliant with regulatory standards like GDPR.

Future Directions

Future research directions may include exploring more diverse datasets beyond the Ubuntu IRC corpus, to generalize findings across different conversational domains and platforms. Furthermore, extending the diagnostic pipeline to compare various state-of-the-art LLMs could yield a comprehensive understanding of their capabilities and limitations in MPCs.

Integrating advanced network science metrics to model conversation structures and experimenting with novel generative techniques for improved summarization and description could also enhance performance in privacy-preserving settings. This research offers a foundational framework upon which further advancements in the field of MPC understanding can be built.

In conclusion, the authors provide an insightful diagnostic tool tailored for MPCs, balancing structural and linguistic complexities. This work not only advances the field of LLMs but also underscores the importance of cohesive modeling strategies that respect data privacy and structural intricacies.