Neighbor-aware Policy Optimization (NAPO)

• NAPO is a decentralized MARL approach for adaptive traffic signal control that integrates neighbor-aware attention and a privileged local critic.
• It employs PPO-style clipped optimization with auxiliary queue prediction to improve training stability and address partial observability.
• NAPO parallels decentralized natural policy gradient methods by localizing value estimation and policy updates under spatial constraints.

Searching arXiv for the named method and its closest antecedents to ground the article in the relevant papers. Neighbor-aware Policy Optimization (NAPO) is a decentralized multi-agent policy-optimization method introduced within the CoordLight framework for network-wide traffic signal control. In that setting, NAPO is presented as a neighbor-aware extension of IPPO that integrates an attention mechanism over adjacent agents, a privileged local critic that conditions on neighbor actions, PPO-style clipped optimization, and auxiliary queue prediction, with the aim of improving coordination under partial observability and interdependent dynamics (Zhang et al., 25 Mar 2026). In a broader methodological sense, NAPO also admits a close conceptual connection to decentralized natural policy-gradient methods that replace globally coupled policy-evaluation and regression steps with neighborhood-restricted approximations under spatial decay assumptions; the paper "Dimension-Free Rates for Natural Policy Gradient in Multi-Agent Reinforcement Learning" formalizes this closely related neighbor-localized natural-gradient paradigm for cooperative MARL (Alfano et al., 2021). By contrast, the acronym NAPO is also used in a different literature for "Negative-aware Preference Optimization" in LLM-based recommendation, which is distinct in both objective and mechanism (Ding et al., 13 Aug 2025).

1. Definition and scope

In CoordLight, NAPO is an advanced MARL algorithm for adaptive traffic signal control that is explicitly designed to improve intra-neighborhood traffic by enhancing decision-making at individual junctions and coordination with neighboring agents (Zhang et al., 25 Mar 2026). Its defining premise is that decentralized traffic control is difficult because each intersection observes only a partial, local view of the traffic system, while the consequences of one intersection’s phase choice propagate to adjacent intersections through queues, spillback, and arrivals. The method therefore makes both the policy and the critic neighbor-aware.

The paper characterizes NAPO as a system with two principal components: an attention-based spatio-temporal actor and a privileged local critic that additionally models neighbors’ actions (Zhang et al., 25 Mar 2026). The actor is intended to determine which neighboring intersections matter most at a given moment, while the critic estimates value in a representation that is conditioned not only on local and neighboring state but also on what neighbors are doing. This produces a policy-update signal that is more tightly coupled to the actual local-interaction structure than plain independent PPO.

A closely related but differently framed line of work appears in decentralized natural policy gradient for cooperative MARL. There, the global policy factorizes as

$$\pi_\theta(a\mid s)=\prod_{k\in K}\pi_{\theta_k}(a_k\mid s),$$

and the central challenge is that direct application of standard NPG makes the Fisher matrix and regression variables global objects whose dimension scales with the number of agents $K$ (Alfano et al., 2021). The resulting decentralized algorithm localizes the value, $Q$-function, and advantage to radius-$r$ neighborhoods and performs local natural-gradient-like updates. This is not named NAPO in that paper, but it is described as very close in spirit to what the label "Neighbor-aware Policy Optimization" suggests (Alfano et al., 2021).

This dual usage establishes two senses of the term. In the strict nomenclature of the traffic-control literature, NAPO denotes the CoordLight algorithm (Zhang et al., 25 Mar 2026). In a broader conceptual sense, it can also denote neighbor-restricted policy optimization in cooperative MARL under local communication constraints, especially when the update geometry and information flow are localized (Alfano et al., 2021).

2. Problem setting in decentralized traffic control

The immediate application domain of NAPO is adaptive traffic signal control over road-network graphs composed of intersections acting as agents (Zhang et al., 25 Mar 2026). Each intersection has a finite action space consisting of collision-free traffic phases, and the learning problem is decentralized: agents act using local observations, but their decisions interact through the traffic network.

CoordLight defines the local neighborhood of agent $i$ as

$\mathcal{H}_i = \{i\} \cup \mathcal{N}^{(1)}_i,$

where $\mathcal{N}^{(1)}_i$ are direct neighbors at shortest-path distance $1$ (Zhang et al., 25 Mar 2026). The actor explicitly uses directional neighbors, writing the observation as

$$z_i(t) = (s_i^t, s_{i_N}^t, s_{i_S}^t, s_{i_E}^t, s_{i_W}^t),$$

so that the ego intersection considers up to four immediate neighbors: North, South, East, and West (Zhang et al., 25 Mar 2026). The paper does not define an explicit physical communication radius; neighbor awareness is instead based on the graph neighborhood of directly connected intersections.

The broader CoordLight framework combines NAPO with Queue Dynamic State Encoding (QDSE), whose state is

$$S(t) = [Q(t), N_{in}(t), N_{out}(t), N_{r}(t), D_{fr}(t), N_{fr}(t)].$$

Per lane, these variables correspond to stopped vehicles or queue length, entering vehicles, departing vehicles, moving vehicles, distance from queue end to foremost moving vehicle, and the number of vehicles following the foremost moving vehicle within a threshold distance (Zhang et al., 25 Mar 2026). QDSE is designed to capture both current congestion and impending congestion, while NAPO uses that state to learn selective dependence on neighboring intersections.

This local-neighborhood framing has a methodological parallel in decentralized NPG. There, each agent $K$0 interacts over an undirected graph $K$1 with distance $K$2, and the objective is to maximize the network-wide average value

$K$3

with

$K$4

That formulation likewise couples a centralized control objective to decentralized communication constraints (Alfano et al., 2021). This suggests that NAPO’s traffic-control instantiation belongs to a larger class of neighborhood-restricted policy-optimization methods for cooperative networked systems.

3. Core architecture and neighbor-awareness mechanisms

NAPO’s actor is not a plain MLP. It uses multi-head attention to determine how much each neighbor should influence the ego agent’s policy (Zhang et al., 25 Mar 2026). The concatenated ego and neighbor observations are first projected to a latent tensor

$K$5

and a state positional vector is appended to identify which node each embedding corresponds to. For each attention head, the query is taken from the ego feature,

$K$6

while keys and values are formed from the four neighbors,

$K$7

The attention weights are computed by scaled dot-product attention, and the output is a weighted sum that is concatenated across heads and combined with the ego feature through a residual connection to form the neighbor-aggregated feature vector $K$8 (Zhang et al., 25 Mar 2026).

The actor is also temporal. After spatial aggregation, the feature is processed by a GRU to integrate temporal dependencies, track evolving traffic conditions, and help the policy react to changing congestion patterns over time (Zhang et al., 25 Mar 2026). The final hidden feature is then passed to a linear layer with softmax to output the policy, alongside an auxiliary linear head for next-step queue-length prediction. The auxiliary prediction is used as a representation-learning aid rather than directly for action selection.

The critic is more expressive than the actor because it explicitly models neighbor action dependency. It is described as an encoder-decoder architecture (Zhang et al., 25 Mar 2026). The state encoder applies the same embedding and spatial aggregation modules as the actor to produce a state-level representation $K$9. The decoder then incorporates neighboring actions by embedding them into

$Q$0

combining them with an action positional vector, and performing another attention step with the encoded state feature as query. The resulting representation is combined via residual connection with the state feature, followed by a GRU and final linear heads that output a value estimate and next-step queue-length prediction (Zhang et al., 25 Mar 2026).

The paper writes the neighbor-aware policy and value as

$Q$1

with augmented observation and action sequences

$Q$2

where $Q$3 and $Q$4 are learned neighbor-weight vectors (Zhang et al., 25 Mar 2026). These learned weights are the mechanism by which influential neighbors are emphasized and irrelevant ones are suppressed.

The closest structural analogue in decentralized NPG is the substitution of global objects by localized ones. For each agent $Q$5, the method defines localized value, $Q$6-function, and advantage,

$Q$7

and solves a local least-squares regression over local score features before performing a neighborhood-restricted update (Alfano et al., 2021). In both formulations, neighbor awareness enters through deliberate restriction of information flow and update computation to local interaction structure.

4. Optimization objective, advantage estimation, and update rules

NAPO adopts PPO-style clipped optimization. The one-step neighbor-aware temporal-difference term is written as

$Q$8

with explicit dependence on augmented observation and neighbor-action context in the critic (Zhang et al., 25 Mar 2026). Generalized Advantage Estimation is then applied: $Q$9 where $r$0 controls the bias-variance tradeoff.

For an individual agent, the PPO-style clipped objective is

$r$1

with importance ratio

$r$2

(Zhang et al., 25 Mar 2026). Entropy regularization is added as

$r$3

and the critic is trained by TD MSE,

$r$4

Both actor and critic also carry auxiliary queue-prediction losses,

$r$5

(Zhang et al., 25 Mar 2026).

The resulting actor and critic objectives average these terms across agents: $r$6

$r$7

(Zhang et al., 25 Mar 2026). Algorithmically, the training pipeline follows PPO-like on-policy training: initialize actor and critic parameters, collect trajectories over $r$8 steps, choose actions with the actor, evaluate with the critic using current state and neighbor actions, store trajectories, compute GAE advantages and TD targets, and update actor and critic for $r$9 epochs with Adam (Zhang et al., 25 Mar 2026).

The decentralized NPG literature uses a different optimization geometry but exhibits a related policy-update structure. Standard NPG starts from the Fisher matrix

$i$0

and update

$i$1

or equivalently a regression in score-feature space followed by an update proportional to the fitted coefficient $i$2 (Alfano et al., 2021). The localized multi-agent version replaces the global regression by local least-squares problems for each agent and updates only neighborhood parameters. This suggests a methodological distinction within the broader NAPO label: CoordLight’s NAPO is PPO-based (Zhang et al., 25 Mar 2026), whereas the closest antecedent in cooperative MARL is natural-gradient-based (Alfano et al., 2021).

5. Relation to localized natural policy gradient and dimension-free decentralization

The paper "Dimension-Free Rates for Natural Policy Gradient in Multi-Agent Reinforcement Learning" provides the clearest theoretical analogue to neighbor-aware policy optimization in the sense of local-information policy updates under decentralized communication constraints (Alfano et al., 2021). Its starting point is that direct application of centralized NPG in a multi-agent network incurs two difficulties: the regression step over the global state-action space suffers a curse of dimensionality, and solving it requires global communication.

To overcome this, the paper imposes a spatial decay of correlation assumption on the dynamics through a Dobrushin-type condition,

$i$3

and an analogous decay assumption for the policy, together with a local policy design constraint ensuring that each agent can compute its policy and its gradient from local information within radius $i$4 (Alfano et al., 2021). Under these assumptions, the local $i$5-function and local value function exhibit exponential decay: $i$6

This decay makes it possible to replace the global advantage by a localized surrogate $i$7, solve a local least-squares regression,

$i$8

and perform the update

$i$9

(Alfano et al., 2021). The local Fisher-like matrix

$\mathcal{H}_i = \{i\} \cup \mathcal{N}^{(1)}_i,$0

plays the role of localized geometry.

The main finite-time guarantee is dimension-free in the sense that there is no explicit dependence on the number of agents $\mathcal{H}_i = \{i\} \cup \mathcal{N}^{(1)}_i,$1 in the convergence rate, while the dependence on action cardinality is logarithmic in $\mathcal{H}_i = \{i\} \cup \mathcal{N}^{(1)}_i,$2 and the state space does not appear explicitly (Alfano et al., 2021). The additional cost of decentralization is a localization error that decays exponentially with communication radius $\mathcal{H}_i = \{i\} \cup \mathcal{N}^{(1)}_i,$3. The paper labels the final additive term in the bound as the localization error and gives an explicit pointwise bound

$\mathcal{H}_i = \{i\} \cup \mathcal{N}^{(1)}_i,$4

up to the notation used in the statement of the result (Alfano et al., 2021).

This theoretical development is not the same algorithm as CoordLight’s NAPO, but it provides a rigorous formulation of the same general principle: neighborhood-restricted policy optimization can preserve favorable global optimization behavior when influence decays with graph distance. A plausible implication is that the CoordLight version of NAPO can be interpreted as an application-specific, PPO-based realization of a broader locality principle that had already been formalized in natural-gradient terms.

6. Empirical results, implementation details, and limitations

CoordLight evaluates NAPO on three real-world traffic datasets composed of up to 196 intersections and reports superior performance across diverse traffic networks with varying traffic flows (Zhang et al., 25 Mar 2026). The paper states that CoordLight with NAPO achieves the best overall travel-time performance across all datasets, including the large-scale New York network, and that it outperforms CoLight, MPLight, Advanced-CoLight, Advanced-MPLight, DenseLight, and SocialLight (Zhang et al., 25 Mar 2026).

The ablation study attributes specific effects to NAPO’s components. Removing the spatio-temporal network causes the model to converge quickly to a suboptimal solution, which the authors interpret as evidence for the importance of spatial and temporal neighbor dependency modeling (Zhang et al., 25 Mar 2026). Removing the critic’s action decoder harms performance, indicating that neighbor action information in the critic improves training stability and overall control quality. The full NAPO version substantially outperforms the IPPO baseline, and the paper explicitly states that integrating action histories into the value estimate accelerates and stabilizes training (Zhang et al., 25 Mar 2026).

The reported training hyperparameters are batch size $\mathcal{H}_i = \{i\} \cup \mathcal{N}^{(1)}_i,$5, actor learning rate $\mathcal{H}_i = \{i\} \cup \mathcal{N}^{(1)}_i,$6, critic learning rate $\mathcal{H}_i = \{i\} \cup \mathcal{N}^{(1)}_i,$7, value loss coefficient $\mathcal{H}_i = \{i\} \cup \mathcal{N}^{(1)}_i,$8, entropy coefficient $\mathcal{H}_i = \{i\} \cup \mathcal{N}^{(1)}_i,$9, prediction loss coefficient $\mathcal{N}^{(1)}_i$0, discount factor $\mathcal{N}^{(1)}_i$1, GAE factor $\mathcal{N}^{(1)}_i$2, PPO clip ratio $\mathcal{N}^{(1)}_i$3, PPO epochs $\mathcal{N}^{(1)}_i$4, and hidden dimensions $\mathcal{N}^{(1)}_i$5 (Zhang et al., 25 Mar 2026). The method uses homogeneous policies, one shared policy for all intersections in a map, and if neighbor information is incomplete, missing entries are padded with zeros and a mask is used so the attention module does not attend to missing data (Zhang et al., 25 Mar 2026).

The limitations stated by the authors are specific and consequential. Experiments use homogeneous road networks with regular intersection structure, so generalization to heterogeneous layouts is not directly demonstrated (Zhang et al., 25 Mar 2026). The study primarily controls signal phases with fixed duration, though the authors note that the method could be adapted to other action spaces such as phase duration control or keep/change decisions. QDSE is evaluated in simulators with noise-free state extraction; robustness is tested with synthetic Gaussian noise, but real deployment may be harder. NAPO also focuses on direct neighbors and local interactions rather than explicitly modeling arbitrarily long-range dependencies except through repeated local propagation. Finally, the queue prediction model is approximate and may underestimate true $\mathcal{N}^{(1)}_i$6, although the authors argue that the error is small for short phases (Zhang et al., 25 Mar 2026).

The decentralized NPG work exhibits complementary limitations and strengths. It requires standard assumptions on spatial decay of correlations for transition dynamics and localized policy structure; under these assumptions, the localization error is exponentially small, and local stochastic projected gradient descent achieves a statistical rate of order $\mathcal{N}^{(1)}_i$7 for the local regression subproblem under bounded score norm (Alfano et al., 2021). This contrast highlights a difference in evidentiary style: CoordLight provides domain-specific empirical validation in large traffic networks (Zhang et al., 25 Mar 2026), whereas the natural-gradient paper provides abstract finite-time guarantees for scalable cooperative MARL under decay conditions (Alfano et al., 2021).

7. Terminological ambiguity and common misconceptions

The acronym NAPO is not unique. In recommendation, "On Negative-aware Preference Optimization for Recommendation" defines NAPO as "Negative-aware Preference Optimization," a reference-free preference-optimization framework for LLM-based sequential recommenders (Ding et al., 13 Aug 2025). Its mechanisms are in-batch negative sharing and dynamic reward margin adjustment, and its task is to exploit negative samples more effectively in recommendation. This usage is unrelated to neighbor-aware decentralized MARL.

The potential confusion is amplified by the fact that the recommendation paper also uses local sharing across similar sequences in a batch, which can make the term "neighbor-aware" appear superficially plausible (Ding et al., 13 Aug 2025). However, the paper explicitly states that the correct expansion is negative-aware rather than neighbor-aware. In encyclopedic treatment, this distinction is essential because the objectives, mathematical formulations, and application domains differ entirely: recommendation preference optimization over hybrid negative sets in one case (Ding et al., 13 Aug 2025), decentralized traffic control with neighbor-conditioned actor-critic learning in the other (Zhang et al., 25 Mar 2026).

A second misconception is that any policy optimization method using neighboring information is equivalent to NAPO. The evidence is more specific. CoordLight’s NAPO is a neighbor-aware extension of IPPO with attention-based spatio-temporal modeling and a privileged critic conditioned on neighbor actions (Zhang et al., 25 Mar 2026). The decentralized NPG paper is very close in spirit and can reasonably be described as a NAPO-style method, but it frames the method as a decentralized natural policy gradient algorithm with local Fisher/regression geometry and decay-of-correlation guarantees rather than using the name NAPO (Alfano et al., 2021).

Taken together, the literature supports a precise formulation. In its strict named sense, Neighbor-aware Policy Optimization denotes the CoordLight algorithm for decentralized traffic signal control (Zhang et al., 25 Mar 2026). In a broader research taxonomy, it denotes a family resemblance among decentralized policy-optimization methods that preserve policy-gradient structure while restricting information flow, value estimation, or optimization geometry to graph-local neighborhoods under communication constraints (Alfano et al., 2021).