Runaway Feedback Loops in Predictive Policing (1706.09847v3)

Abstract: Predictive policing systems are increasingly used to determine how to allocate police across a city in order to best prevent crime. Discovered crime data (e.g., arrest counts) are used to help update the model, and the process is repeated. Such systems have been empirically shown to be susceptible to runaway feedback loops, where police are repeatedly sent back to the same neighborhoods regardless of the true crime rate. In response, we develop a mathematical model of predictive policing that proves why this feedback loop occurs, show empirically that this model exhibits such problems, and demonstrate how to change the inputs to a predictive policing system (in a black-box manner) so the runaway feedback loop does not occur, allowing the true crime rate to be learned. Our results are quantitative: we can establish a link (in our model) between the degree to which runaway feedback causes problems and the disparity in crime rates between areas. Moreover, we can also demonstrate the way in which \emph{reported} incidents of crime (those reported by residents) and \emph{discovered} incidents of crime (i.e. those directly observed by police officers dispatched as a result of the predictive policing algorithm) interact: in brief, while reported incidents can attenuate the degree of runaway feedback, they cannot entirely remove it without the interventions we suggest.

• The paper develops a formal urn-based model that quantifies how iterative police deployments amplify data-driven biases.
• Empirical simulations show that relying on discovered incidents creates self-reinforcing loops that skew resource allocation.
• The study proposes intervention strategies using reported incidents to counteract feedback loops and more accurately reflect true crime rates.

The paper presents a rigorous mathematical analysis of predictive policing systems, focusing on how iterative decision-making based on discovered crime data can lead to severe feedback loops. The work formalizes the interaction between the placement of police resources and the recorded evidence of crime, particularly emphasizing the distinction between discovered incidents (direct observations following dispatch) and reported incidents (complaints or contextual reports from residents).

Key contributions and findings include:

• Mathematical Framework: The authors develop a formal model that captures the dynamics of predictive policing. By representing the process as an urn model, the analysis exposes how sampling bias can be amplified over repeated iterations. Under the model, the probability of selecting an area for policing is influenced by prior discovered incident counts, which in turn are functions of historical police deployments. This recursive mechanism mathematically explains why certain neighborhoods encounter repeated deployments even if the underlying crime rates are not proportionately high.
• Runaway Feedback Characterization: The analysis quantitatively links the extent of runaway feedback to the disparity in true crime rates between areas. More specifically, if one region has even a slightly elevated crime rate, the iterative update rule can escalate police presence disproportionately, effectively masking the true relative differences in crime. Such runaway feedback loops are shown to be a direct consequence of using discovered incident data without corrective measures.
• Empirical Evidence via Simulations: Extensive simulations using the urn-based experiments demonstrate how feedback loops develop over time. The simulation results underscore that when discovered incidents are used as the primary input for updating the policing model, the feedback mechanism can become self-reinforcing. In contrast, incorporating reported incidents (which are less influenced by the allocation process) can attenuate but not entirely eliminate the bias. This empirical evidence solidifies the theoretical findings by providing clear numerical illustrations of how disproportionate police attention can occur.
• Intervention Strategies: The paper also proposes interventions that adjust the system inputs in a black-box manner. These adjustments are designed to counter the runaway dynamics by modifying the influence of discovered incidents relative to reported ones. The proposed interventions are shown, through both theoretical and simulation-based evaluations, to enable the true underlying crime rate to be learned more accurately by mitigating the self-reinforcing bias.
• Quantitative Insights: The paper provides quantitative bounds for the severity of feedback loops under various conditions. The interplay between discovered and reported crime is elucidated, highlighting that while reported data can temper the bias to an extent, it cannot completely counterbalance the feedback without structural modifications to the predictive policing mechanism.

For a veteran researcher, the paper offers a compelling account of the inherent challenges in deploying online learning systems in high-stakes environments like law enforcement. It is notable for its blend of rigorous theoretical analysis backed by simulations, providing a clear pathway from abstract model formulation to practical implications. The quantitative demonstrations, particularly those relating the degree of feedback to crime rate disparities, are highly relevant for understanding and mitigating algorithmic bias in predictive policing environments.