REFER: Mitigating Bias in Opinion Summarisation via Frequency Framed Prompting

Abstract: Individuals express diverse opinions, a fair summary should represent these viewpoints comprehensively. Previous research on fairness in opinion summarisation using LLMs relied on hyperparameter tuning or providing ground truth distributional information in prompts. However, these methods face practical limitations: end-users rarely modify default model parameters, and accurate distributional information is often unavailable. Building upon cognitive science research demonstrating that frequency-based representations reduce systematic biases in human statistical reasoning by making reference classes explicit and reducing cognitive load, this study investigates whether frequency framed prompting (REFER) can similarly enhance fairness in LLM opinion summarisation. Through systematic experimentation with different prompting frameworks, we adapted techniques known to improve human reasoning to elicit more effective information processing in LLMs compared to abstract probabilistic representations.Our results demonstrate that REFER enhances fairness in LLMs when summarising opinions. This effect is particularly pronounced in larger LLMs and using stronger reasoning instructions.