Measuring the Quality of Answers in Political Q&As with Large Language Models (2404.08816v5)

Abstract: This article proposes a new approach for assessing the quality of answers in political question-and-answer sessions. We measure the quality of an answer based on how easily and accurately it can be recognized in a random set of candidate answers given the question's text. This measure reflects the answer's relevance and depth of engagement with the question. Like semantic search, we can implement this approach by training a LLM on the corpus of observed questions and answers without additional human-labeled data. We showcase and validate our methodology within the context of the Question Period in the Canadian House of Commons. Our analysis reveals that while some answers have a weak semantic connection to questions, hinting at some evasion or obfuscation, they are generally at least moderately relevant, far exceeding what we would expect from random replies. We also find a meaningful correlation between answer quality and the party affiliation of the members of Parliament asking the questions.

• The paper introduces a novel method using semantic search to assess answer quality in political Q&As.
• It applies a fine-tuned BERT model with contrastive learning and cosine similarity to measure semantic relevance.
• Empirical findings reveal partisan and topic-based variations, highlighting strategic differences in political responses.

Novel Measurement of Answer Quality in Political Discourses Using LLMs

Introduction

This research extensively evaluates answer quality within the Canadian political context, specifically during the "Question Period" in the House of Commons. The approach centralizes on the novelty of utilizing semantic search capabilities to correlate the quality of a response to the ability to infer the corresponding question, introducing a methodological innovation to the assessment of political discourse.

Methodological Framework

The paper deploys a LLM fine-tuned on a comprehensive dataset spanning several legislative sessions from 2006 to 2021. This fine-tuning process involves sentence embeddings from a BERT-based model, specifically tailored to assess semantic similarities between parliamentary questions and corresponding answers.

Key Aspects of the Model:

1. Semantic Search Paradigm: Uses the capability of the model to search for answers that are semantically close to the input questions.
2. Self-Supervised Learning: Employs contrastive learning where the model is trained to distinguish between correct answers and a set of potential but incorrect answers.
3. Cosine Similarity Measure: A metric used to quantify the quality of answers based on the semantic closeness to the original questions.

Empirical Findings

The analysis of over 58,000 parliamentary exchanges provided robust insights into the dynamics of question handling based on party affiliations and the topics discussed.

• Party Affiliation: The quality of answers varied significantly with the party of the member posing the question. Notably, members of the ruling party or ideologically similar parties received higher-quality responses.
• Question Topics: Topics such as government accountability, ethics, and budget management often received lower-quality responses, suggesting a strategic avoidance or obfuscation in politically sensitive areas.

Statistical Observations:

• The model demonstrated a skewed distribution of cosine similarities indicating a variation in answer relevance.
• High-quality responses were notably prevalent in discussions related to less politically charged topics or those aligned with the government's ideological stance.

Theoretical Implications

This paper contributes to the scholarly understanding of political communication by providing a computational method to objectively analyze the quality of political discourse. It ties the operational definition of answer quality to the practical ability to reconstruct the original question from the answer, thereby aligning theoretical concepts with implementable metrics.

Practical Applications

Beyond academic implications, the methodological approach suggested has practical applications in real-time monitoring of political debates, providing non-partisan assessments of political communication quality, and could potentially extend to other forms of public discourse analysis.

Future Research Directions

Future studies might explore cross-national comparisons using similar legislative frameworks or extend this model to other forms of political communication such as debates, interviews, or press briefings. Additionally, adapting the model's training to include more diverse or ideologically varied datasets could enhance its applicability and robustness across different political contexts.

Conclusion

This research marks a significant step towards integrating advanced NLP techniques with political science research, offering a novel lens through which to assess the functionality of one of the parliamentary democracy's core elements—accountability through dialogue.