Shapley Q-value Framework

• Shapley Q-value framework is a mathematical and algorithmic approach leveraging Shapley values to decompose the Q-function into individual contributions in RL.
• It enforces key axioms from cooperative game theory—efficiency, symmetry, dummy, and additivity—to ensure fair and interpretable value attributions.
• The framework applies Monte Carlo estimation for scalable computation and has demonstrated efficacy in various domains such as navigation, games, and multi-agent systems.

The Shapley Q-value framework is a mathematical and algorithmic formalism that leverages the Shapley value from cooperative game theory to attribute the contribution of individual components—state features or agents—to the action-value function (Q-function) in both single-agent and multi-agent reinforcement learning (RL) settings. This approach yields rigorous, axiomatic, and interpretable decompositions of value predictions or rewards, facilitating explanation, credit assignment, and analysis of policies and learning outcomes.

1. Formal Definition and Shapley Attribution

Let $F = \{1, \ldots, n\}$ denote the index set of state features for a single-agent RL task. For a fixed state $s = (s_1, \ldots, s_n)$ and action $a$, the framework defines a coalitional game $(F, v)$, where the characteristic function $v : 2^F \to \mathbb{R}$ is given by

$$v(C) \equiv \overline{Q}_C(s, a) \equiv \mathbb{E}[Q(S, a) \mid S_C = s_C],$$

i.e., the expected Q-value conditioned on knowing only the features in $C$, marginalizing over the empirical distribution $p^\pi(S)$ induced by the agent's policy. The Shapley Q-value attribution for feature $i$ is: $$\phi_i(s, a) = \sum_{C \subseteq F \setminus \{i\}} \frac{|C|! \, (n - |C| - 1)!}{n!} \left( v(C \cup \{i\}) - v(C) \right).$$ Here, $v(C \cup \{i\}) - v(C)$ quantifies the marginal contribution of feature $i$ to a coalition $C$, rigorously aggregating over all orderings weighted by the standard Shapley coefficients. For multi-agent global-reward games, the analogous construction applies to agents and coalitions, with the marginal Q-value of an agent defined relative to strategic coalitional policies (Wang et al., 2019, Wang et al., 2021).

2. Axiomatic and Theoretical Foundations

The Shapley Q-value satisfies the classical four axioms of Shapley value in this RL context (Beechey et al., 12 May 2025, Wang et al., 2019):

• Efficiency: $$\sum_{i \in F} \phi_i(s, a) = Q(s, a) - \overline{Q}_\emptyset(s, a)$$, where $\overline{Q}_\emptyset(s, a)$ is the baseline Q-value under full marginalization.
• Symmetry: If two features $i$, $j$ are exchangeable in all coalitions, then $\phi_i = \phi_j$.
• Dummy: If a feature $i$ never affects the Q-value (i.e., $v(C \cup \{i\}) = v(C)$ for all $C$), then $\phi_i = 0$.
• Additivity: For sum of games $v + u$, the attributions sum: $\phi_i(v+u) = \phi_i(v) + \phi_i(u)$.

In the multi-agent setting, Markov Shapley values (MSVs) further extend these properties to action-based credit assignment, leading to unique, fair, and interpretable decompositions (Wang et al., 2021).

3. Algorithmic Realization and Complexity

Exact evaluation of Shapley Q-values requires $O(2^n n)$ calls per state-action pair, which is tractable only for small $n$. Monte Carlo estimation via random permutations is standard:

Input: state s, action a, feature-set F={1..n}, Q-function Q, sampler pπ(.|S_C=s_C), M = # permutations For each feature i in F: φ̂_i ← 0 For m = 1..M: Draw random permutation O of F C ← ∅; prev_val ← E[Q(S,a)] For each feature i in O in order: Sample s' ~ pπ(S | S_C = s_C) cur_val ← Q(s', a) φ̂_i ← φ̂_i + (cur_val − prev_val) prev_val ← cur_val C ← C ∪ {i} φ̂_i ← φ̂_i / M for all i Return φ̂_1,…,φ̂_n

Each permutation requires $O(n)$ conditional samples, leading to a total complexity of $O(M n)$, where $M \approx 10^3$–$10^4$ yields stable estimates (Beechey et al., 12 May 2025). In MARL, similar sampling applies to agent coalitions, and deep RL architectures (e.g., SQDDPG, SHAQ) use neural critics to approximate marginal contributions efficiently (Wang et al., 2019, Wang et al., 2021).

4. Illustrative Examples and Empirical Insights

The framework provides principled attributions across diverse RL domains:

• Road-sign navigation: Features like "Direction" and "Distance" have comparable Shapley attributions, elucidating their joint necessity for optimal navigation value predictions (Beechey et al., 12 May 2025).
• Dice re-roll: Shapley Q-value distinguishes between "keep" and "re-roll" actions by assigning negative attributions to dice dragging down the expected Q-value, yielding clear quadrant separability over input states.
• Taxi domain: Dramatic shifts in attribution (e.g., for "passenger-on-taxi" near drop-off) reveal phase transitions in agent incentive structure.

In multi-agent settings, Shapley Q-value approaches yield agent-specific, coalition-averaged reward signals:

• Predator–Prey and SMAC: SHAQ assigns credit proportional to actual individual impact, accurately identifying dummy agents and rewarding pivotal contributors (Wang et al., 2021).
• Cooperative Navigation, Traffic Junction: SQDDPG demonstrates improved credit assignment and faster convergence compared to shared-reward or counterfactual baselines (Wang et al., 2019).
• Interpretability: Under optimal actions, agents' Shapley Q-values approach equal division (reflecting efficiency and fairness), while under suboptimality, attributions reflect true marginal contribution.

5. Unification with Prior and Contemporary Approaches

The Shapley Q-value framework strictly subsumes previous RL-Shapley explanation methods. Notably:

• Methods applying SHAP with fully marginal feature distributions ignore feature dependencies and violate Shapley axioms in the RL context.
• Explanations targeting only $V(s)$ or $\pi(s,a)$ lack the generality and unification provided by conditioning on $Q$ (Beechey et al., 12 May 2025).
• The Shapley Q framework rigorously incorporates conditional sampling, axiomatic uniqueness, and consistent Monte Carlo estimation, thereby reconciling behavior, performance, and value explanations under one cooperative-game-based formalism.
• In multi-agent domains, Shapley Q-value methods (SHAQ, SQDDPG) outperform additive (VDN), monotonic (QMIX), and counterfactual (COMA) baselines in both theoretical fairness and empirical precision (Wang et al., 2021, Wang et al., 2019).

6. Extensions, Limitations, and Practical Considerations

The framework supports several extensions and practical implementations:

• Conditional Sampling: High-fidelity sampling of $p^\pi(S | S_C=s_C)$ is essential; this can be achieved via empirical replay buffers or generative models.
• Scalability: While Monte Carlo estimation is tractable for moderate $n$, very high-dimensional settings may require further model approximation.
• Generalization: The cooperative-game abstraction allows analogous Shapley value attributions for policy probabilities (behavior explanation) and value functions.
• Multi-agent credit assignment: Shapley Q-value assignments are robust to agent symmetry and team structure, always lying within the (stochastic) core of the value distribution problem.
• Open Challenges: Ensuring conditional independence for feature/agent sampling, compressing attribution computation in deep RL, and relaxing the assumption of convex characteristic functions remain ongoing research directions.

By treating value estimation as a cooperative game among observed features or agents and enforcing strict Shapley axioms, the Shapley Q-value framework underpins interpretable and mathematically sound attributions of value and reward in reinforcement learning and multi-agent systems (Beechey et al., 12 May 2025, Wang et al., 2019, Wang et al., 2021).