The Moral Integrity Corpus: A Benchmark for Ethical Dialogue Systems

Abstract: Conversational agents have come increasingly closer to human competence in open-domain dialogue settings; however, such models can reflect insensitive, hurtful, or entirely incoherent viewpoints that erode a user's trust in the moral integrity of the system. Moral deviations are difficult to mitigate because moral judgments are not universal, and there may be multiple competing judgments that apply to a situation simultaneously. In this work, we introduce a new resource, not to authoritatively resolve moral ambiguities, but instead to facilitate systematic understanding of the intuitions, values and moral judgments reflected in the utterances of dialogue systems. The Moral Integrity Corpus, MIC, is such a resource, which captures the moral assumptions of 38k prompt-reply pairs, using 99k distinct Rules of Thumb (RoTs). Each RoT reflects a particular moral conviction that can explain why a chatbot's reply may appear acceptable or problematic. We further organize RoTs with a set of 9 moral and social attributes and benchmark performance for attribute classification. Most importantly, we show that current neural LLMs can automatically generate new RoTs that reasonably describe previously unseen interactions, but they still struggle with certain scenarios. Our findings suggest that MIC will be a useful resource for understanding and LLMs' implicit moral assumptions and flexibly benchmarking the integrity of conversational agents. To download the data, see https://github.com/GT-SALT/mic

• The paper introduces MIC, a dataset of 38K prompt-reply pairs and 99K moral Rules of Thumb to benchmark ethical dialogue systems.
• It outlines a robust data collection and annotation framework based on r/AskReddit posts and responses from prominent AI models like BlenderBot and GPT-Neo.
• Models using T5, GPT-2, and BART achieved strong performance in generating and classifying moral attributes, advancing AI moral reasoning.

The Moral Integrity Corpus: A Benchmark for Ethical Dialogue Systems

The paper "The Moral Integrity Corpus: A Benchmark for Ethical Dialogue Systems" introduces the Moral Integrity Corpus (MIC), which is a comprehensive dataset designed to facilitate ethical benchmarking in conversational AI. This corpus aims to address the challenge of moral reasoning in open-domain dialogue systems by providing structured annotations for AI-generated responses based on moral "Rules of Thumb" (RoTs).

Introduction and Motivation

Conversational AI systems, while promising in many domains such as education, healthcare, and customer support, often face challenges related to trust and moral integrity. Models can inadvertently generate responses that are insensitive or morally questionable, which can undermine user trust. MIC provides a systematic resource to understand the moral intuitions and judgments embedded in AI dialogues. It comprises 38,000 prompt-reply pairs annotated with 99,000 distinct RoTs, each reflecting different moral convictions that justify the acceptance or rejection of a chatbot's reply.

Figure 1: A representative MIC annotation. We evaluate the AI response (Reply) to a human query (Prompt) using Rules of Thumb (RoT), which describe ``right and wrong'' ways to handle the conversation.

Data Collection and Annotation Framework

The paper outlines the process of collecting and annotating dialogue data. The prompt-reply pairs are sourced from r/AskReddit posts, enriched with responses from leading AI chatbots, including BlenderBot and GPT-Neo. Filtering techniques ensured these responses included normative content. The annotation scheme is inspired by applied ethics, where annotators draft RoTs that capture moral reasoning associated with AI responses. RoTs are further categorized based on alignment with the response, consensus on rules, violation severity, and moral foundations such as care, fairness, and liberty.

RoT Generation and Attribute Classification

The authors trained models to automatically describe moral assumptions present in AI responses. They employed LLMs like T5, GPT-2, and BART to generate RoTs for unseen interactions, achieving strong performance metrics, particularly with T5 under beam search.

Figure 2: Our forward language modeling setup for RoT Generation.

Moreover, attribute classifiers were built to categorize RoTs based on violation severity, global consensus, and moral foundations, with models achieving above-human performance consistency.

Challenges and Limitations

The paper identifies unique challenges in modeling moral reasoning in dialogues, such as handling nuanced moral viewpoints, unexpected moral violations arising from chatbot limitations, and adversarial probing by users that force the system into compromising replies.

Figure 3: (Left) \% of annotators who align with the given political leaning. (Right) \% of annotations written by annotators with the given political leaning.

While the dataset is biased by annotator demographics (primarily United States-based individuals), MIC serves as a robust benchmark for understanding and moderating conversational AI systems based on diverse moral perspectives.

Future Directions and Conclusion

MIC provides a foundation for developing computational models capable of reasoning about dialogue system integrity, guiding research towards enhanced moderation frameworks and ethical AI deployment. Future work could expand on cultural and demographic variances, exploring dynamic moral reasoning. Ultimately, MIC aims to support the development of conversational agents that respect diverse moral constructs, enhancing trust and user experience in AI systems.

The corpus and models presented offers a significant resource for researchers aiming to advance conversational AI towards moral competence, aligning AI-generated dialogue with human ethical standards.