Attention-Aware Inverse Planning

• Attention-aware inverse planning is a framework that infers latent selective attention rather than solely optimizing for rewards.
• It integrates multiple formulations such as policy, temporal, and spatial attention to recover internal biases and representation constraints.
• These methods improve performance in multi-agent tasks, automated driving simulations, and human maze planning by yielding more accurate behavioral inferences.

Attention-aware inverse planning denotes a family of inverse planning and inverse reinforcement learning formulations in which the latent explanatory structure is not limited to reward or goal identity, but includes selective attention: over candidate futures and time, over goals encoded inside a policy, or over the subset of objects and relations that enter the internal representation used for planning. In this sense, attention can be treated as a learned relevance filter, an explicit mental state, or a constrained construal policy, and inverse inference aims to recover that attentional structure from behavior rather than explaining all deviations solely by altered rewards or noise (Rosbach et al., 2020, Long et al., 2024, Banerjee et al., 29 Oct 2025).

1. Conceptual scope and relation to standard inverse planning

Standard inverse planning and IRL typically infer latent goals, preferences, or reward parameters from observed actions under known dynamics. Attention-aware inverse planning changes the explanatory target. In "Inverse Attention Agents" the latent state is a vector of attention weights over gradient-field goals rather than a reward function or a discrete goal label; in "Planning on the fast lane" attention is a mechanism for learning where to focus over sampled trajectories and over planning cycles when inferring reward; in "Estimating cognitive biases with attention-aware inverse planning" the inverse problem is to recover attentional bias parameters $\lambda$ under known rewards and dynamics (Long et al., 2024, Rosbach et al., 2020, Banerjee et al., 29 Oct 2025).

A recurring point of contrast is that ignoring attention can induce systematic misidentification of goals, competence, or preferences. The 2025 cognitive-bias formulation states this explicitly: if behavior is generated by a resource-limited construal process, then fitting a fully informed planner can misinfer goals and competence. The 2024 multi-agent formulation makes an analogous move inside a Markov game, where the relevant latent variable is an agent’s current weighting over candidate goals. The driving IRL formulation retains reward inference, but attention determines which candidate futures and which portions of temporal context dominate that inference (Banerjee et al., 29 Oct 2025, Long et al., 2024, Rosbach et al., 2020).

The main formulations can be compared as follows.

Paradigm	Attentional state	Inverse target
Path-integral IRL	Policy attention over sampled trajectories; temporal attention over reward histories	Situation-dependent reward weights
Inverse Attention Agents	Continuous attention weights over gradient-field goals	Another agent’s attentional state
Cognitive-bias AAIP	Construal $C$ and bias parameters $\lambda$	Attentional bias parameters
Spotlight-VGC	Local spatial attention over nearby obstacles	Mechanistic constraint on representation

Taken together, these works suggest that attention-aware inverse planning is not a single formalism. It is a broader pattern in which a forward planning model contains an explicit or implicit attentional bottleneck, and inverse inference targets that bottleneck rather than treating all behavioral variation as reward variation.

2. Formalizations of attentional latent state

In path-integral IRL for automated driving, the planning problem is posed as a continuous-control MDP with a Gibbs distribution over sampled trajectories,

$$p(\pi \mid \bm{\theta}) = \frac{1}{Z}\exp(-\bm{\theta}^\top \bm{f}^{\pi}), \quad Z = \sum_{\pi \in \Pi} \exp(-\bm{\theta}^\top \bm{f}^{\pi}).$$

The inverse problem is still reward inference, but attention enters twice. Policy attention constructs a context vector

$\mathbf{z} = \sum_{i=1}^N \alpha_i h_i,$

where $\alpha_i$ is a soft distribution over sampled policies, and temporal attention produces a mixture of previously inferred rewards,

$$\bm{\theta}^{(t+1)} = \sum_{h \in H} w_h \, \bm{\theta}^{(h)}.$$

The inverse process therefore becomes attention-aware over both policy space and time: the learned reward is a function of which futures and which history segments are emphasized (Rosbach et al., 2020).

In the cognitive-bias formulation, the state is object-oriented and the key latent variable is a construal $C\subseteq O$, the subset of objects represented for planning. The forward model defines a construed policy

$\pi_C(a \mid s) = \pi^*(a \mid s[C]),$

a value of representation

$\text{VOR}(s, C) = V(s, \pi_C) + \text{Cost}(C),$

and a bias-modulated construal selection rule

$C$0

The inverse problem is then

$C$1

with $C$2. This differs from standard IRL because reward $C$3 and dynamics $C$4 are treated as known, while the latent variable is an attentional bias governing representation (Banerjee et al., 29 Oct 2025).

In multi-agent systems, attention itself becomes the explicit mental state. Each agent encodes its observation as goal-like gradient-field features $C$5, and its policy expresses a vector of attention weights

$C$6

The forward policy is

$C$7

and the inverse module is trained to infer attention weights from observations,

$C$8

Here, attention weights function as the agent’s explicit mental state about priorities over candidate goals, so inverse planning is recast as attention-state inference rather than reward recovery (Long et al., 2024).

A related but mechanistically distinct formulation appears in the maze-planning study of value-guided construal. There the forward model is not a full inverse-planning implementation, but it introduces an attentional spotlight that modifies which obstacles are available to the planning representation. With spotlight width $C$9, the model replaces static obstacle relevance by a local spatial average,

$\lambda$0

This provides a concrete forward mechanism by which attention constrains the internal model over which planning operates (Castanheira et al., 11 Jun 2025).

3. Algorithmic realizations

The driving formulation implements a two-level attention architecture inside maximum-entropy path-integral IRL. PACNN encodes sampled policies with 1D CNNs, average pooling, and fully connected layers, then applies policy attention to produce a low-dimensional context vector for reward prediction. PTACNN adds a 2-layer LSTM, a 4-layer fully connected network, and a final softmax over past planning cycles, yielding a mixture of past reward functions rather than a de novo reward estimate at every cycle. Training combines the MaxEnt IRL log-likelihood with a semi-supervised attention loss that encourages higher attention on policies closer to the demonstration under a weighted Euclidean metric (Rosbach et al., 2020).

The multi-agent formulation uses three stages: a self-attention policy, an inverse attention network, and an attention-updating controller. Phase 1 trains the Self-Att policy with MAPPO for 20M steps while collecting $\lambda$1 pairs. Phase 2 retains the latest 10% of this dataset and trains $\lambda$2 offline with L2 loss, using $\lambda$3, batch size $\lambda$4, hidden dim $\lambda$5, 3000 epochs, and early stopping via validation. Phase 3 replaces the policy by the Inverse-Att architecture, initializes the update module $\lambda$6 to identity on self-weights and zero on others, and trains again with MAPPO for 20M steps. The architecture is modular rather than jointly optimized: the inverse-attention module is trained offline and then frozen before the final RL stage (Long et al., 2024).

The cognitive-bias formulation has two regimes. In tabular DrivingWorld, construals are enumerable, $\lambda$7 and $\lambda$8 can be computed exactly, and $\lambda$9 is obtained by dynamic programming, after which off-the-shelf gradient-based optimizers in SciPy maximize the log-likelihood over $$p(\pi \mid \bm{\theta}) = \frac{1}{Z}\exp(-\bm{\theta}^\top \bm{f}^{\pi}), \quad Z = \sum_{\pi \in \Pi} \exp(-\bm{\theta}^\top \bm{f}^{\pi}).$$0. In GPUDrive with Waymo scenes, the method pre-trains a generalist policy $$p(\pi \mid \bm{\theta}) = \frac{1}{Z}\exp(-\bm{\theta}^\top \bm{f}^{\pi}), \quad Z = \sum_{\pi \in \Pi} \exp(-\bm{\theta}^\top \bm{f}^{\pi}).$$1 via PPO, defines construals by masking inputs, estimates $$p(\pi \mid \bm{\theta}) = \frac{1}{Z}\exp(-\bm{\theta}^\top \bm{f}^{\pi}), \quad Z = \sum_{\pi \in \Pi} \exp(-\bm{\theta}^\top \bm{f}^{\pi}).$$2 by Monte Carlo rollouts, and treats the three-dimensional bias inference problem as black-box optimization solved by Bayesian optimization. Computing behavioral utilities of all construals, 166 construals across 10 scenes with 40 rollouts each, takes about an hour on a GPU, while inference over $$p(\pi \mid \bm{\theta}) = \frac{1}{Z}\exp(-\bm{\theta}^\top \bm{f}^{\pi}), \quad Z = \sum_{\pi \in \Pi} \exp(-\bm{\theta}^\top \bm{f}^{\pi}).$$3 for 80 trajectories takes a few minutes on a CPU (Banerjee et al., 29 Oct 2025).

The maze-planning work contributes a mechanistic attentional module rather than a standalone inverse planner. Static VGC first scores obstacles by their contribution to the utility-complexity trade-off of a construal. The spotlight-VGC extension then locally smooths those scores according to spatial proximity. Because this smoothing is defined over the representational variables that enter planning, it provides a forward model of how attentional overspill can alter the latent representation that inverse planning would need to explain (Castanheira et al., 11 Jun 2025).

4. Attention, Theory of Mind, and cognitive bias

In the multi-agent setting, attention-aware inverse planning is explicitly linked to Theory of Mind. The model is positioned as a cognitive ToM system focused on attention rather than beliefs or desires: agent $$p(\pi \mid \bm{\theta}) = \frac{1}{Z}\exp(-\bm{\theta}^\top \bm{f}^{\pi}), \quad Z = \sum_{\pi \in \Pi} \exp(-\bm{\theta}^\top \bm{f}^{\pi}).$$4 infers agent $$p(\pi \mid \bm{\theta}) = \frac{1}{Z}\exp(-\bm{\theta}^\top \bm{f}^{\pi}), \quad Z = \sum_{\pi \in \Pi} \exp(-\bm{\theta}^\top \bm{f}^{\pi}).$$5’s attentional state by “putting itself in $$p(\pi \mid \bm{\theta}) = \frac{1}{Z}\exp(-\bm{\theta}^\top \bm{f}^{\pi}), \quad Z = \sum_{\pi \in \Pi} \exp(-\bm{\theta}^\top \bm{f}^{\pi}).$$6’s shoes,” computing $$p(\pi \mid \bm{\theta}) = \frac{1}{Z}\exp(-\bm{\theta}^\top \bm{f}^{\pi}), \quad Z = \sum_{\pi \in \Pi} \exp(-\bm{\theta}^\top \bm{f}^{\pi}).$$7, then concatenates inferred other-attention with its own attention and updates its control weights through $$p(\pi \mid \bm{\theta}) = \frac{1}{Z}\exp(-\bm{\theta}^\top \bm{f}^{\pi}), \quad Z = \sum_{\pi \in \Pi} \exp(-\bm{\theta}^\top \bm{f}^{\pi}).$$8. The result is a planning architecture that conditions action selection on inferred attentional states of others of the same type (Long et al., 2024).

In the cognitive-bias formulation, attention is not merely a weight over observations. It is a structured selection of which objects enter the construed state at all. Biases $$p(\pi \mid \bm{\theta}) = \frac{1}{Z}\exp(-\bm{\theta}^\top \bm{f}^{\pi}), \quad Z = \sum_{\pi \in \Pi} \exp(-\bm{\theta}^\top \bm{f}^{\pi}).$$9 are heuristic tendencies to include or exclude certain object types or geometric relations, such as ice patches, cones, parked cars, or the driving features Deviation from Ego Heading, Relative Heading, and Deviation from Ego Collision. The framework therefore separates three explanatory layers: the known task reward, the cognitive cost of representation, and the bias parameters that shape construal choice. This makes it possible to model an agent as rational given its construal while still appearing systematically biased from the perspective of a fully informed observer (Banerjee et al., 29 Oct 2025).

The maze-planning study provides human behavioral evidence for the forward side of this picture. It reports strong proximity effects in awareness ratings: for closest neighbours, rank 1 and rank 2, coefficients are positive in Ho 1, with $\mathbf{z} = \sum_{i=1}^N \alpha_i h_i,$0 and $\mathbf{z} = \sum_{i=1}^N \alpha_i h_i,$1, while furthest neighbours, ranks 5 and 6, are about $\mathbf{z} = \sum_{i=1}^N \alpha_i h_i,$2. The paper describes this as attentional overspill. It also reports that the interaction between static VGC relevance and lateralization is positive, with Ho 1 giving $\mathbf{z} = \sum_{i=1}^N \alpha_i h_i,$3 and $\mathbf{z} = \sum_{i=1}^N \alpha_i h_i,$4, and that stronger attention effects predict more sparse representations, with Spearman $\mathbf{z} = \sum_{i=1}^N \alpha_i h_i,$5 around $\mathbf{z} = \sum_{i=1}^N \alpha_i h_i,$6 to $\mathbf{z} = \sum_{i=1}^N \alpha_i h_i,$7. These findings support the claim that attention determines what is available to planning, not just how efficiently a fixed representation is read out (Castanheira et al., 11 Jun 2025).

A nearby but distinct extension is "Acting as Inverse Inverse Planning." That work does not model attention explicitly, but it optimizes actions to manipulate an observer’s inverse-planning posterior over goals, alignment, and rationality. This suggests an adjacent perspective: once a planner or audience model is specified, attention-aware inverse planning can be complemented by methods that actively steer what an observer infers and when, rather than merely recovering latent attentional state from behavior (Chandra et al., 2023).

5. Empirical domains and findings

In multi-agent continuous 2D Markov games based on MPE, inverse attention agents improve ad hoc teamwork and mixed-motive performance. In the full results table, Spread rises from $\mathbf{z} = \sum_{i=1}^N \alpha_i h_i,$8 for MAPPO and $\mathbf{z} = \sum_{i=1}^N \alpha_i h_i,$9 for Self-Att to $\alpha_i$0 for Inverse-Att; Navigation rises from $\alpha_i$1 and $\alpha_i$2 to $\alpha_i$3; Adversary–sheep improves from $\alpha_i$4 and $\alpha_i$5 to $\alpha_i$6; and Grassland–sheep improves from $\alpha_i$7 and $\alpha_i$8 to $\alpha_i$9. The inverse network’s rank-order recovery of attention is also strong: top-1 attention prediction is close to 100% accuracy, and top-2 is also very high accuracy. Human experiments show the same direction of effect, with Spread agent reward $$\bm{\theta}^{(t+1)} = \sum_{h \in H} w_h \, \bm{\theta}^{(h)}.$$0 for Inverse-Att versus $$\bm{\theta}^{(t+1)} = \sum_{h \in H} w_h \, \bm{\theta}^{(h)}.$$1 for Self-Att and $$\bm{\theta}^{(t+1)} = \sum_{h \in H} w_h \, \bm{\theta}^{(h)}.$$2 for MAPPO, and with Adversary–sheep improving to $$\bm{\theta}^{(t+1)} = \sum_{h \in H} w_h \, \bm{\theta}^{(h)}.$$3 from $$\bm{\theta}^{(t+1)} = \sum_{h \in H} w_h \, \bm{\theta}^{(h)}.$$4 and $$\bm{\theta}^{(t+1)} = \sum_{h \in H} w_h \, \bm{\theta}^{(h)}.$$5 (Long et al., 2024).

In automated driving, attention improves IRL efficiency and temporal stability. PACNN matches 1DCNN and Bi1DCNN on EVD and ED while using about $$\bm{\theta}^{(t+1)} = \sum_{h \in H} w_h \, \bm{\theta}^{(h)}.$$6 fewer parameters, and PTACNN(+S) achieves the best test ED and OPD, with average OPD $$\bm{\theta}^{(t+1)} = \sum_{h \in H} w_h \, \bm{\theta}^{(h)}.$$7 versus $$\bm{\theta}^{(t+1)} = \sum_{h \in H} w_h \, \bm{\theta}^{(h)}.$$8 for 1DCNN. Qualitative visualizations show that policy attention focuses on collision-free, human-like trajectories, while temporal attention remains stable during persistent interactions and shifts rapidly during context switches such as approaching stop signs or merge opportunities (Rosbach et al., 2020).

In bias estimation with Waymo-derived scenarios, the method is evaluated on 215 synthetic agents generated from 10 real-world scenes. For each agent, 80 trajectories, about 12 minutes of driving, are generated from the biased construal-selection model. Scatter plots of true $$\bm{\theta}^{(t+1)} = \sum_{h \in H} w_h \, \bm{\theta}^{(h)}.$$9 versus inferred $C\subseteq O$0 show tight correspondence for DfEH, RH, and DfEC, and mean squared error decreases with more data and stabilizes around 12 minutes of driving. In the tabular comparison with IRL, the best IRL model still yields significantly worse negative log-likelihood than the attention-aware model, 265 versus 513 and above, and produces qualitatively different patterns of risk-taking (Banerjee et al., 29 Oct 2025).

In human maze planning, spotlight-VGC improves model fit over static VGC in all datasets, with $C\subseteq O$1 in Ho 1, $C\subseteq O$2 in Ho 2, and $C\subseteq O$3 overall in dSC 1; in dSC 1 the improvement is concentrated in non-lateralized mazes, where $C\subseteq O$4. This is consistent with the claim that attention matters most when task-relevant information does not align with natural attentional contours (Castanheira et al., 11 Jun 2025).

The inverse-inverse-planning study supplies complementary evidence that planning-based observer models can predict human judgments. In the kitchen domain, inverse inverse planning yields 73% target-consistent “helping” responses versus 6% for naive planning, 62% “hindering” versus 29%, and 73% “indifferent” versus 75%. In the hill domain, a light box optimized to mime heaviness is chosen as heavier than a light box 95.7% of the time and even as heavier than a genuinely heavy box 68.6% of the time (Chandra et al., 2023). While these results are not themselves attention inference, they show that planning-based latent-state models can shape human interpretation at a fine temporal granularity.

6. Limitations, misconceptions, and adjacent directions

A central misconception is that attention-aware inverse planning is simply standard IRL with an added attention layer. The surveyed work shows several non-equivalent formulations. Some infer reward with attention over trajectories and time; some infer an explicit attention vector as the mental state; some infer attentional biases under known reward and dynamics; some model only the forward attentional constraint on representation. Attention-aware inverse planning is therefore better understood as a class of inverse models in which selective representation is itself part of the latent explanation.

The limitations are correspondingly heterogeneous. In the multi-agent formulation, inverse inference is same-type only, $C\subseteq O$5 assumes a fixed number of teammates, the environment is fully observable, $C\subseteq O$6 outputs a point estimate rather than explicit uncertainty, and training is staged rather than joint. In the driving IRL formulation, generating about 2500 policies per cycle is computationally significant, demonstrations come from simulation, rewards depend on hand-crafted features, and attention is scalar and single-head. In the cognitive-bias formulation, rewards and dynamics are assumed known, a single construal is chosen at episode start, the bias function is linear in hand-designed features, human validation in the driving domain is pending, and there is no joint reward-bias inference (Long et al., 2024, Rosbach et al., 2020, Banerjee et al., 29 Oct 2025).

The maze-planning work makes a different limitation explicit: it does not yet implement full Bayesian inverse planning over attentional states, but instead provides empirical and computational constraints for such a model, including spotlight width, lateralization, and overspill. A plausible implication is that future attention-aware inverse planners in human behavioral domains will benefit from combining trajectories with awareness reports and eye-tracking, because the study shows that awareness ratings and gaze-constrained control analyses contain information about the attentional field that shaped planning (Castanheira et al., 11 Jun 2025).

A further adjacent direction is the reinterpretation of brain-inspired planning as an inverse-planning substrate. The thesis "Brain-Inspired Planning for Better Generalization in Reinforcement Learning" develops top-down attention, spatial abstraction, goal-conditioned value and distance models, and a feasibility evaluator. It does not implement a full inverse planner, but it argues that these are the ingredients needed to infer what an agent is trying to do from what it attends to and how it chooses sub-goals. This suggests a route toward attention-aware inverse planning in hierarchical RL: combine intention-conditioned attention $C\subseteq O$7, goal-conditioned forward models, and feasibility filtering to infer latent subgoals while ruling out infeasible explanations (Zhao, 9 Nov 2025).

In summary, attention-aware inverse planning is best characterized by a shift in explanatory ontology. Instead of assuming that observed behavior should be explained only by latent rewards, it treats selective attention, construal, and representational bias as first-class latent causes. Across multi-agent Theory of Mind, path-integral IRL for driving, value-guided construal in human planning, and cognitive-bias estimation, the unifying idea is that planning depends on what is represented, and inverse planning must therefore infer not only what an agent wants, but also what it is effectively attending to.