People Can Accurately Predict Behavior of Complex Algorithms That Are Available, Compact, and Aligned

Abstract: Users trust algorithms more when they can predict the algorithms' behavior. Simple algorithms trivially yield predictively accurate mental models, but modern AI algorithms have often been assumed too complex for people to build predictive mental models, especially in the social media domain. In this paper, we describe conditions under which even complex algorithms can yield predictive mental models, opening up opportunities for a broader set of human-centered algorithms. We theorize that users will form an accurate predictive mental model of an algorithm's behavior if and only if the algorithm simultaneously satisfies three criteria: (1) cognitive availability of the underlying concepts being modeled, (2) concept compactness (does it form a single cognitive construct?), and (3) high alignment between the person's and algorithm's execution of the concept. We evaluate this theory through a pre-registered experiment (N=1250) where users predict behavior of 25 social media feed ranking algorithms that vary on these criteria. We find that even complex (e.g., LLM-based) algorithms enjoy accurate prediction rates when they meet all criteria, and even simple (e.g., basic term count) algorithms fail to be predictable when a single criterion fails. We also find that these criteria determine outcomes beyond prediction accuracy, such as which mental models users deploy to make their predictions.

• The paper establishes the ACA theory, demonstrating that human prediction accuracy hinges on algorithmic availability, compactness, and alignment.
• It uses a pre-registered experiment with 1250 participants assessing 25 social media feed algorithms varying in ACA compliance.
• The study finds ACA-compliant algorithms achieve high prediction accuracy (mean 85%), while violations yield near-chance performance.

Predictability of Complex Algorithms: The ACA Theory

Introduction and Theoretical Motivation

The paper "People Can Accurately Predict Behavior of Complex Algorithms That Are Available, Compact, and Aligned" (2601.18966) investigates the long-standing tension between algorithmic complexity and human predictability in AI systems, with a particular focus on social media feed algorithms. Contrary to the prevalent view that only simple algorithms are predictable by users, the authors formalize a theory—Availability, Compactness, and Alignment (ACA)—which posits that people can form accurate predictive mental models even of highly complex algorithms, provided these three cognitive criteria are satisfied.

Availability refers to whether the core algorithmic concept is cognitively accessible—users can recognize what is being modeled or optimized. Compactness demands that the concept underlying the algorithm can be distilled into a single cohesive construct, amenable to cognitive chunking. Alignment requires that the user’s understanding of the concept matches the algorithm’s operationalization, so their simulation leads to similar outputs.

The ACA theory is formulated as necessary and sufficient for predictive mental model formation. The proposition is that unless all three criteria are met, users’ ability to predict an algorithm’s behavior will be at chance, regardless of the computational simplicity or opacity of the algorithm’s internals.

Figure 1: ACA theory — the intersection of Availability, Compactness, and Alignment is necessary for human predictability, regardless of algorithmic complexity.

Methodological Framework

The authors conducted an extensive pre-registered experiment (N=1250) using 25 social media feed ranking algorithms designed to systematically vary in their degree of satisfaction of ACA criteria. These included algorithms ranging from trivial (e.g., likes, reverse chronological order) to complex (LLM-based toxicity or writing quality rankings), and those purposely violating individual ACA criteria.

The experimental procedure comprised exposure, training, and testing phases. Participants first observed ranked feeds reflecting a specific algorithm’s outputs. Training involved ranking post pairs with feedback, facilitating candidate mental model refinement. The final testing measured prediction accuracy without feedback.

Figure 2: Experimental measurement of prediction accuracy—participants iteratively trained and tested on ranking tasks derived from their inferred mental models of the feed algorithm.

Empirical Findings

Statistical analyses established that user prediction accuracy was significantly elevated (mean 85%) for algorithms meeting all three ACA criteria. Any violation of a single criterion resulted in performance near chance (mean 54%), supporting the ACA theory’s necessity and sufficiency claims.

Notably, algorithms that were complex in implementation (e.g., deep learning models for toxicity) yet ACA-compliant were predicted as accurately as the simplest ones. In contrast, algorithms that were computationally trivial but violated compactness or alignment (e.g., random logic, noisy feature combinations) were unpredictable to users.

Figure 3: Participant prediction accuracy peaks only for algorithms satisfying Availability, Compactness, and Alignment; all other configurations perform at or near baseline.

Figure 4: Detailed accuracy breakdown—ACA algorithms approach 80%+, while all others cluster within 10% of guessing.

Mental Model Formation and Feature Reliance

Analysis of participants' self-reported mental models substantiated the link between ACA compliance and accurate algorithmic prediction. In ACA conditions, most users formed mental models matching the underlying algorithm’s specification and achieved high accuracy.

Even when incorrect mental models correlated with available features (e.g., wrongly assuming engagement was critical), users sometimes achieved above-chance prediction due to collinearity between features (likes, views, retweets, etc.).

Figure 5: ACA conditions had higher proportions of mental models matching design intent, though notable exceptions (e.g., writing quality) highlight within-category variation.

Figure 6: Engagement-related features dominated mental model descriptions, demonstrating a strong prior and availability bias, even when algorithmic relevance was absent.

Implications for Algorithm Design and Interpretability

The findings have several practical and theoretical implications:

• Human-Centered Algorithm Design: The ACA framework provides actionable guidelines for engineering systems whose global behavior is legible to users even when the internal operations are strongly opaque or complex, enabling trust and appropriate reliance in AI-assisted contexts.
• Participatory and Transparent Systems: ACA compliance can be viewed as a testable criterion before algorithm deployment, offering a rigorous basis for participatory design and regulatory oversight.
• Folk Theory Formation and Tradeoffs: When ACA criteria are not met, users employ available and compact but poorly aligned mental models or maintain fragmented heuristics, which do not lead to reliable prediction but help them interact with the system in uncertain contexts.
• Interpretability Beyond Simplicity: The study challenges the prevailing narrative that interpretability is synonymous with architectural simplicity; cognitive simplicity—anchored in ACA—is the key determinant.

Nuances and Future Directions

The authors acknowledge several boundary conditions—most notably, the domain specificity of ACA criteria (validated here in social media feeds) and possible gradations rather than binary satisfaction of ACA factors. The mental models are population-dependent and change with expertise, social context, and priming.

Additional research is encouraged to generalize ACA across AI-driven decision domains (hiring, recommendation, healthcare), and to develop quantitative measures of ACA factors. Classifying algorithms for ACA can be challenging in real-world, personalized, or evolving contexts where the concepts or their combinations shift over time.

Conclusion

The ACA theory provides a precise conceptual framework for the relationship between cognitive accessibility and human predictability of algorithmic systems. Empirical evidence supports the view that even the most complex algorithms can be made predictable if they instantiate available, compact, and aligned concepts. This reframes interpretability as a property emergent from cognitive mapping rather than computational transparency.

The implications for AI governance, HCI, and algorithmic design are substantial—opening paths to create and validate advanced systems that both maximize performance and human predictability, with direct consequences for trust, fairness, and user empowerment.