Improving Human Performance with Value-Aware Interventions: A Case Study in Chess

Abstract: AI systems are increasingly used to assist humans in sequential decision-making tasks, yet determining when and how an AI assistant should intervene remains a fundamental challenge. A potential baseline is to recommend the optimal action according to a strong model. However, such actions assume optimal follow-up actions, which human decision makers may fail to execute, potentially reducing overall performance. In this work, we propose and study value-aware interventions, motivated by a basic principle in reinforcement learning: under the Bellman equation, the optimal policy selects actions that maximize the immediate reward plus the value function. When a decision maker follows a suboptimal policy, this policy-value consistency no longer holds, creating discrepancies between the actions taken by the policy and those that maximize the immediate reward plus the value of the next state. We show that these policy-value inconsistencies naturally identify opportunities for intervention. We formalize this problem in a Markov decision process where an AI assistant may override human actions under an intervention budget. In the single-intervention regime, we show that the optimal strategy is to recommend the action that maximizes the human value function. For settings with multiple interventions, we propose a tractable approximation that prioritizes interventions based on the magnitude of the policy-value discrepancy. We evaluate these ideas in the domain of chess by learning models of humans from large-scale gameplay data. In simulation, our approach consistently outperforms interventions based on the strongest chess engine (Stockfish) in a wide range of settings. A within-subject human study with 20 players and 600 games further shows that our interventions significantly improve performance for low- and mid-skill players while matching expert-engine interventions for high-skill players.

• The paper introduces a formal framework leveraging human policy-value discrepancies to determine optimal intervention states in sequential decision-making.
• It employs behavioral cloning and simulation on large-scale chess datasets, demonstrating statistically significant improvements for low-skill players.
• The study highlights that adapting AI interventions to individual human behavior can mitigate error compounding and enhance overall decision quality.

Value-Aware AI Interventions in Sequential Decision-Making: Analysis and Implications

Problem Formulation and Motivation

The paper presents a formal framework for value-aware interventions in sequential decision-making tasks, specifically addressing the limitations of action recommendations based solely on expert models. Traditional AI-aided approaches frequently employ optimal policy recommendations—such as Stockfish moves in chess—under the assumption of optimal continuation by the human user. However, humans often deviate from optimal play, resulting in downstream states that can be disproportionately challenging, reducing overall utility compared to more human-aligned alternatives. The authors operationalize this insight by leveraging discrepancies between human behavioral policies and their associated value functions in the context of Markov Decision Processes (MDPs), identifying states where intervention may improve expected outcomes.

Utilizing large-scale human behavioral datasets, the authors train models that approximate both the human policy $\pi_H$ and value function $V^{\pi_H}$, enabling data-driven identification of intervention opportunities. The framework constrains interventions using a budget parameter $B$, reflecting practical limits imposed by cognitive load, autonomy, or system design.

Theoretical Contributions

The authors derive closed-form optimal strategies for the single-intervention regime: for a fixed human policy, the optimal override action at a state is the one maximizing the human value function $Q^{\pi_H}(s,a)$. The improvement from intervention is quantified as $$\Delta^{\pi_H}(s,a) = Q^{\pi_H}(s,a) - V^{\pi_H}(s)$$. In multiple-intervention settings, a tractable heuristic is introduced where states with largest policy-value discrepancies are prioritized, under the approximation that intervention shifts the trajectory only minimally away from the human policy for small $B$.

This approach sharply contrasts with prior baselines, including optimal-action strategies which neglect the impact of suboptimal future human actions [bastani2026improving]. The explicit accounting for downstream policy divergence is a key methodological innovation.

Empirical Evaluation in Chess

Simulation Analysis

The framework is evaluated in chess, a domain offering rich sequential complexity, well-defined rules, large-scale gameplay data, and robust optimal-play engines. The authors train a behavioral cloning model (based on Leela T82), parameterized on player rating, to encapsulate both move distribution and expected outcomes.

Simulation results show the value-aware intervention ("Valuemax") consistently outperforms both human baselines and optimal move recommendations (Stockfish) in single-intervention settings. Notably, for low-skill players (800 rating), Valuemax is more than 2% better than Stockfish, while the advantage converges to 0.3% at expert levels (2400 rating). In multiple-intervention settings, Valuemax remains superior for low intervention budgets; Stockfish overtakes as intervention frequency increases and the effective policy diverges from $\pi_H$. These results quantitatively validate the theoretical prediction that policy-value discrepancies encode valuable intervention opportunities.

Analysis of Chess Concepts

Descriptive analysis utilizing Stockfish evaluation features reveals that interventions for low-skill players correlate strongly with exposed opponent kings, whereas high-skill players exhibit less correlation. This suggests that value-aware interventions adaptively target structural weaknesses tailored to skill level. Such analyses support explainable AI and could inform future work on interpretable intervention logic.

Human-Subject Study

A within-subject study involving 20 chess players and 600 games corroborates simulation findings. Valuemax interventions produced statistically significant performance improvements for low- and mid-skill participants, with greater than 35% advantage at the lowest skill tier, while performance for high-skill players was indistinguishable from optimal recommendations. These results reinforce the practical relevance of accounting for downstream human policy and highlight stratification of intervention benefit by user skill.

Implications and Future Directions

Practical Implications

The results underscore the need for AI assistants in sequential, high-stakes domains to adjust recommendations based on modeled user behavior—not merely return optimal actions. Systems that overlook the human's policy can inadvertently degrade user performance, especially when interventions are infrequent. Value-aware strategies thus hold promise for domains such as clinical decision support, programming, and financial trading, where error compounding and state-space complexity are prevalent.

Theoretical Implications

The formalism bridges reinforcement learning theory with human behavioral modeling by exploiting policy-value inconsistencies as actionable intervention signals. This advances principled approaches to cooperative AI design, pending advances in behavioral modeling for heterogeneous, nonstationary policies.

Speculation on Future Developments

Future work could address:

• Extension to Heterogeneous Domains: Applying value-aware interventions to environments with limited human data or greater behavioral heterogeneity, possibly via adaptive or online learning.
• Partial Adherence Modeling: Relaxing the assumption of full recommendation compliance, integrating models for partial adherence [grand2026best].
• Skill Development Over Time: Studying interventions that not only optimize immediate outcomes but facilitate learning and skill acquisition.
• Explainable and Concept-Based Interventions: Designing assistant logic based on human-understandable concepts to improve transparency and pedagogical efficacy.
• Real-time Deployment and Mixed-AI Policies: Engineering practical systems for dynamic intervention selection under real-world constraints, adaptive to user-specific behavioral distributions.

Conclusion

This paper rigorously advances value-aware AI intervention strategies for sequential decision-making, empirically demonstrating their superiority over optimal-action recommendations in complex environments such as chess—especially for suboptimal human agents and under low intervention budgets. The findings exhort practitioners to explicitly model downstream human policy when designing intervention logic and open avenues for more explainable, adaptive, and pedagogically informed AI systems.