Algorithmic Drift: A Simulation Framework to Study the Effects of Recommender Systems on User Preferences

Abstract: Digital platforms such as social media and e-commerce websites adopt Recommender Systems to provide value to the user. However, the social consequences deriving from their adoption are still unclear. Many scholars argue that recommenders may lead to detrimental effects, such as bias-amplification deriving from the feedback loop between algorithmic suggestions and users' choices. Nonetheless, the extent to which recommenders influence changes in users leaning remains uncertain. In this context, it is important to provide a controlled environment for evaluating the recommendation algorithm before deployment. To address this, we propose a stochastic simulation framework that mimics user-recommender system interactions in a long-term scenario. In particular, we simulate the user choices by formalizing a user model, which comprises behavioral aspects, such as the user resistance towards the recommendation algorithm and their inertia in relying on the received suggestions. Additionally, we introduce two novel metrics for quantifying the algorithm's impact on user preferences, specifically in terms of drift over time. We conduct an extensive evaluation on multiple synthetic datasets, aiming at testing the robustness of our framework when considering different scenarios and hyper-parameters setting. The experimental results prove that the proposed methodology is effective in detecting and quantifying the drift over the users preferences by means of the simulation. All the code and data used to perform the experiments are publicly available.

• The paper introduces a simulation framework that quantifies algorithmic drift by measuring changes in user preferences over time using the ADS and DTC metrics.
• It models interactions between users and recommender systems by incorporating user resistance, inertia, and randomness to simulate feedback loops.
• Experimental results with synthetic datasets demonstrate that variations in user behavior parameters significantly influence the magnitude of algorithmic drift.

Algorithmic Drift: A Simulation Framework to Study the Effects of Recommender Systems on User Preferences

Introduction

The paper "Algorithmic Drift: A Simulation Framework to Study the Effects of Recommender Systems on User Preferences" explores the long-term effects of recommender systems on user preferences. It focuses on the concept of "algorithmic drift", which refers to the tendency of recommendation algorithms to alter user preferences over time. This research addresses the need for a controlled environment to evaluate recommendation algorithms before deployment, aiming to quantify the potential drift in user preferences.

Simulation Framework

The proposed simulation framework models interactions between users and a recommender system, considering user behavior characterized by resistance, inertia, and randomness. The framework accounts for a feedback loop where user choices are both influenced by recommendations and contribute to further recommendations. The simulation iteratively builds a probabilistic graph representing user interactions, providing a detailed representation of user preference evolution.

Figure 1: Framework overview showcasing the simulation process for user interactions with recommender systems.

Metrics for Algorithmic Drift

The paper introduces two metrics to quantify algorithmic drift:

1. Algorithmic Drift Score (ADS): This metric adapts the Random Walk Controversy Score (RWC) to measure the likelihood of users consuming content from a target category compared to others. It captures the probability distribution of user pathways over time.
2. Delta Target Consumption (DTC): This percentage rate measures the change in user consumption of a target category before and after interacting with the recommender system. It quantifies the increase in interaction with specific content types.

Experimental Evaluation

The framework's efficacy is demonstrated through synthetic datasets that simulate varying proportions of non-radicalized, semi-radicalized, and radicalized users interacting with neutral and harmful content. The experiments evaluate the impact of different user behavior parameters—resistance, inertia, and randomness—on algorithmic drift.

Various settings illustrate the robustness of the framework. For instance, increasing the proportion of semi-radicalized users enhances the drift effect due to collaborative filtering mechanisms. The interplay of resistance and inertia is also examined, showing that lower resistance and higher inertia lead to more significant preference alterations.

Figure 2: Algorithmic Drift Score (ADS) and Delta Target Consumption (DTC), illustrating varying responses to changes in user random factor $\eta$.

Conclusion

The paper presents a comprehensive methodology for simulating the impact of recommendation systems on user preferences over time. The introduction of ADS and DTC metrics provides a robust quantification of algorithmic drift, allowing researchers and practitioners to evaluate the effects of different recommender configurations. Future work could explore dynamic item catalogs, contextual factors, and other phenomena like popularity bias and diversity.

This framework offers valuable insights for the development and deployment of recommender systems, ensuring that potential adverse effects on user preferences are understood and mitigated prior to real-world application.