What We Know About Using Non-Engagement Signals in Content Ranking (2402.06831v1)

Abstract: Many online platforms predominantly rank items by predicted user engagement. We believe that there is much unrealized potential in including non-engagement signals, which can improve outcomes both for platforms and for society as a whole. Based on a daylong workshop with experts from industry and academia, we formulate a series of propositions and document each as best we can from public evidence, including quantitative results where possible. There is strong evidence that ranking by predicted engagement is effective in increasing user retention. However retention can be further increased by incorporating other signals, including item "quality" proxies and asking users what they want to see with "item-level" surveys. There is also evidence that "diverse engagement" is an effective quality signal. Ranking changes can alter the prevalence of self-reported experiences of various kinds (e.g. harassment) but seldom have large enough effects on attitude measures like user satisfaction, well-being, polarization etc. to be measured in typical experiments. User controls over ranking often have low usage rates, but when used they do correlate well with quality and item-level surveys. There was no strong evidence on the impact of transparency/explainability on retention. There is reason to believe that generative AI could be used to create better quality signals and enable new kinds of user controls.

• The paper demonstrates that incorporating non-engagement signals, such as quality metrics and surveys, can enhance user retention beyond traditional engagement metrics.
• It employs rigorous experimental designs to address the complexity of measuring the effects of these alternative signals on content ranking.
• The study underscores the balance between engagement and quality signals, paving the way for more transparent, effective ranking algorithms.

Evaluating Non-Engagement Signals in Content Ranking Systems

The paper "What We Know About Using Non-Engagement Signals in Content Ranking" presents an analytical discourse on the potential benefits of incorporating non-engagement signals into content ranking algorithms used by online platforms. The research, conducted through a workshop with industry and academic experts, strives to outline the propositions that could enhance both platform objectives and societal outcomes via the integration of alternative signals beyond user engagement metrics.

Summary of Findings

The primary takeaway from the paper is the substantiated claim that predicted engagement metrics drive increased user retention. However, the potential for enhanced retention exists when non-engagement signals, such as quality metrics or item-level surveys, are incorporated into the ranking processes. Several conclusions from the workshop point toward the untapped potential and challenges of employing these signals.

• Engagement Metrics vs. Non-Engagement Signals: Traditionally, platforms utilize engagement metrics such as clicks, likes, or shares as predominant elements in their content ranking algorithms. However, evidence indicates that while these metrics increase retention, alternative signals relating to content quality or direct feedback through user surveys can offer long-term retention enhancements.
• Complexity of Measuring Effects: Accurately determining the effects of non-engagement signals is challenging and requires rigorous experimental design. Workshop participants noted the difficulties of estimating the weights of value-model objectives, advocating for more involved data-driven approaches to optimize retention-focused algorithms.
• Quality as a Determinant: Notably, platforms are observed to give significant ranking weight to quality metrics, acknowledging their potential to improve retention. However, a paradox emerges as these quality indicators sometimes exhibit a negative correlation with engagement metrics. This scenario necessitates a nuanced understanding of how both types of metrics affect user experience.
• Surveys and User Controls: Item-level and user-level surveys offer additional layers of insight that engagement metrics alone cannot capture, often aligning with quality outcomes. Meanwhile, user controls, although not widely utilized, provide crucial information aligned with user expectations and intent.

Implications and Future Directions

The implications of using non-engagement signals are manifold, affecting algorithmic transparency, content quality, and user control mechanisms. In domains like health and politics, balancing content quality with engagement is exceptionally pertinent. Workshop discussions also highlighted the nascent role of generative AI in scaling non-engagement efforts, potentially impacting content monitoring and personalization at unprecedented scales.

While the paper adequately captures the current landscape and challenges in implementing non-engagement signals, it posits that future research can further explore integrating these signals with existing ranking systems. There remains a need for comprehensive studies examining the long-term impacts of such implementations on user well-being and content polarization, areas currently under-explored due to data availability and methodological constraints.

Conclusion

This research summarizes the insights drawn from industry best practices, emphasizing a balanced approach in combining both engagement and non-engagement signals for content ranking. The authors advocate for transparency and user-level involvement to create digital ecosystems that cater more effectively to nuanced user preferences and societal expectations. Future developments will likely involve enhanced models of user interaction, aided by sophisticated AI technologies, to overcome existing challenges in content ranking algorithms.