The Intelligent Disobedience Game: Formulating Disobedience in Stackelberg Games and Markov Decision Processes

Abstract: In shared autonomy, a critical tension arises when an automated assistant must choose between obeying a human's instruction and deliberately overriding it to prevent harm. This safety-critical behavior is known as intelligent disobedience. To formalize this dynamic, this paper introduces the Intelligent Disobedience Game (IDG), a sequential game-theoretic framework based on Stackelberg games that models the interaction between a human leader and an assistive follower operating under asymmetric information. It characterizes optimal strategies for both agents across multi-step scenarios, identifying strategic phenomena such as ``safety traps,'' where the system indefinitely avoids harm but fails to achieve the human's goal. The IDG provides a needed mathematical foundation that enables both the algorithmic development of agents that can learn safe non-compliance and the empirical study of how humans perceive and trust disobedient AI. The paper further translates the IDG into a shared control Multi-Agent Markov Decision Process representation, forming a compact computational testbed for training reinforcement learning agents.

• The paper introduces the Intelligent Disobedience Game, a Stackelberg framework in which a partially informed leader proposes actions and a better-informed follower obeys safe instructions or vetoes harmful ones.
• The paper proves that optimal play prevents harm and reaches a goal in finite time whenever a safe path exists, while identifying safety traps that cause indefinite goal denial when no goal is reachable.
• The paper translates the game into coupled leader POMDP and follower MDP models, creating a testbed for multi-agent reinforcement learning, shared-control systems, and studies of trust in disobedient AI.

Overview

The paper introduces the Intelligent Disobedience Game (IDG), a formal game-theoretic framework for modeling situations in which an assistive agent must decide whether to comply with or override a human's instruction to prevent harm. The motivating scenario is the guide-dog paradigm: a handler (leader) proposes actions toward a task goal, while the guide (follower) possesses strictly better information about environmental hazards and may veto proposals. The work makes two contributions: an extensive-form Stackelberg characterization of this interaction, including equilibrium analysis for single-step and finite-horizon variants, and a translation of the game into decoupled shared-control Markov decision processes suitable for multi-agent reinforcement learning and user studies (2603.20994).

The Intelligent Disobedience Game

The IDG is defined as an extensive-form Stackelberg game $$\mathcal{G}_{\text{IDG}} = \langle S, A_L, A_F, T, u_L, u_F \rangle$$. The leader's action space at each state is partitioned into goal-reaching actions $A_g(s)$, harmful actions $A_h(s)$, and other actions $A_o(s)$. The follower's action set is binary: obey or disobey. Disobedience nullifies the proposed action, leaving the state unchanged. Crucially, the information structure is asymmetric: the leader can distinguish only between $A_g(s)$ and $A_h(s) \cup A_o(s)$, whereas the follower distinguishes all three subsets. Payoffs encode the leader's preference for reaching the goal ($+1$) and avoiding harm ($-1$), and the follower's preference for correctly intercepting harmful actions ($+1$ for disobeying a harmful action, $-1$ for disobeying a non-harmful one).

An important structural observation is that, despite the Stackelberg convention favoring the leader who moves first, the follower here holds effective control over execution. The equilibrium analysis resolves whether this apparent power imbalance undermines the leader's objectives.

Equilibrium analysis

For the single-step base case ($A_g(s)$0-IDG), the analysis splits on whether goal-reaching actions exist. When $A_g(s)$1, the follower's optimal strategy is to disobey exactly the harmful actions; anticipating this, the leader's expected payoff is $A_g(s)$2 under any pure or mixed strategy, so every leader strategy profile constitutes an equilibrium. When $A_g(s)$3, the follower's optimal strategy $A_g(s)$4 pairs with any leader strategy supported on $A_g(s)$5 to yield payoffs $A_g(s)$6.

The finite-horizon extension ($A_g(s)$7-IDG) reveals a qualitatively new phenomenon the authors call safety traps: subsets of states in which no goal-reaching action exists, all non-harmful actions remain within the subset, and the subset is closed under reachability via non-harmful actions. A follower that steers play into such a region collects an infinite stream of positive payoff from intercepting harmful proposals while permanently denying the leader its goal — indefinite harm avoidance without goal attainment.

The central result is that safety traps do not survive optimal play when a goal is reachable. By backward induction with mixed strategies over actions that may lead to the goal, the leader can commit to persistent goal-directed proposals; since disobeying non-harmful actions incurs negative payoff for the follower, the follower is pressured into compliance, and entry into traps is avoided. Consequently, if a goal state is reachable, optimal play terminates at the goal in finite time; if not, the game begins inside a safety trap and repeats indefinitely. This yields a clean dichotomy with a practical implication: a rational follower guarantees harm prevention regardless of reachability, and goal attainment whenever a safe path exists — without the follower ever needing a reward signal tied to the goal itself. The authors highlight this as a design simplification: a real-world safety layer need only model hazards, not the human's objective.

MDP formulation

To make the framework amenable to standard MARL algorithms, the IDG is recast as two coupled but individually optimized models sharing state space and transition function: a POMDP $A_g(s)$8 for the partially informed leader and a fully observable MDP $A_g(s)$9 for the follower. The transition function acts as a shared-control operation protocol: obedience executes the leader's proposal; disobedience returns a no-op. Reward functions mirror the game's utilities, with the follower penalized both for obeying into harm and for wrongly vetoing benign proposals. Because optimal policies exist in MDPs, the equilibria derived analytically can be validated empirically, and the formulation doubles as a compact testbed for training RL agents and for structured human-subject experiments on trust in disobedient systems.

Relation to prior work

The IDG extends the lineage of formal models of corrigibility and intervention, most directly the off-switch game of Hadfield-Menell et al., which modeled an agent's incentives to permit shutdown. Where the off-switch game concerns the agent's willingness to be interrupted, the IDG concerns the agent's authority to interrupt, with asymmetric observability as the driver of the interaction. The formulation also connects to constructive AI rebellion literature, which holds that selective disobedience presupposes reliable obedience capacity, and to shared-control architectures in which an operation protocol fuses human and machine inputs into a single executed action. The paper notes, though does not resolve, the connection to theory-of-mind capabilities: assessing risk requires modeling human intent and knowledge, yet LLMs are documented to fail on trivially perturbed ToM tasks, so the informational assumptions embedded in the IDG are nontrivial to satisfy with current language-model-based followers.

Limitations and open questions

The authors are explicit about several assumptions. First, the environment is static: the classification of actions into $A_h(s)$0, $A_h(s)$1, and $A_h(s)$2 does not change over time, whereas realistic deployments such as guide-dog assistance involve dynamically shifting hazard structures. Extending the IDG to environments altered by the agents' own actions or by exogenous conditions is left open. Second, the equilibrium results assume perfect rationality and perfect hazard detection by the follower; the paper concedes that faulty AI systems could obey a harmful action, making safety-trap regions especially dangerous in practice — a failure mode the framework identifies but does not quantify. Third, the base game admits only binary veto; richer interventions, such as the follower proposing alternative actions, are noted as extensions rather than analyzed. Finally, the human-perception side — how trust and transparency evolve when a system repeatedly overrides instructions — is motivated but not studied empirically here.

Conclusion

This paper supplies a mathematical foundation for intelligent disobedience by casting it as a Stackelberg game with asymmetric information and deriving its equilibria. The main findings are that a rational follower optimally vetoes only harmful proposals, that goal attainment is guaranteed under optimal play whenever a safe path exists, and that the otherwise problematic "safety trap" phenomenon is neutralized by committed goal-directed leadership. The accompanying MDP formulation converts these results into a trainable, testable computational artifact. The framework's value going forward depends on relaxing its static-environment and perfect-rationality assumptions and on empirical validation with human participants.