Neighbor-Specific Q-Learning

• Neighbor-Specific Q-Learning is defined as localizing Q-value estimation using neighborhood relations, enabling scalable credit assignment in various RL contexts.
• It covers approaches from geometric nearest neighbors in continuous states to graph-local aggregation and explicit per-neighbor action-value learning in multi-agent systems.
• This locality principle improves computational efficiency and strategic coordination, while balancing approximation errors and scalability trade-offs.

Neighbor-Specific Q-Learning denotes a family of Q-learning formulations in which value estimation and policy improvement are localized by a neighborhood relation rather than defined over a fully global representation. In the literature, that neighborhood relation has several distinct meanings. In continuous-state reinforcement learning, it refers to geometric nearest neighbors in state space, so Q-values are extended or smoothed by nearest-neighbor regression (Shah et al., 2018). In graph-structured multi-agent reinforcement learning, it refers to adjacent agents, local interaction partners, or a fixed communication topology, so Q-updates depend on graph-local state summaries, neighbor-weighted rewards, or neighbor-only regularizers (Mao et al., 2019). In networked control, it refers to a $\kappa$-neighborhood or to neighbor-to-neighbor communication constraints, so critics and updates are computed from local system couplings rather than the full network state (Olsson et al., 2024). The shared principle is locality: neighbor structure constrains approximation, credit assignment, exploration, or communication.

1. Terminological scope and major variants

The term is not used uniformly across the literature. Some papers use it for nonparametric nearest-neighbor smoothing in continuous state spaces, while others use it for graph-local or edge-local value learning in multi-agent systems. A precise reading therefore requires identifying what counts as a “neighbor” in the underlying formulation.

The major usages surveyed here can be summarized as follows.

Neighborhood meaning	Core mechanism	Representative papers
Geometric nearest neighbors in state space	$k$-NN or support-limited nearest-neighbor regression for $Q$ estimation	(Shah et al., 2018, Zhao et al., 2023)
Graph neighbors in MARL	GCN aggregation, neighbor-only KL regularization, or graph-local state encoding	(Mao et al., 2019, Zheng et al., 2024)
Explicit interaction partners	Per-neighbor action values, neighbor-selection policies, or edge-local rewiring values	(Ren et al., 2024, Weng et al., 1 Sep 2025)
Local reward coupling without per-neighbor tables	Neighbor-weighted rewards or locally aggregated reinforcement signals	(Daeichian et al., 2019, Du et al., 29 Jan 2026)
Control-theoretic locality	$\kappa$-local critics or neighbor-to-neighbor distributed updates	(Olsson et al., 2024, Mallick et al., 20 Nov 2025)

A useful distinction is between explicit per-neighbor Q-learning and neighborhood-conditioned Q-learning. In the explicit form, the learner stores objects such as $Q_{i,j}(s,a)$ or separate Q-networks for interaction choices toward particular neighbors. In the conditioned form, Q-values remain indexed only by local state and action, but the state, reward, or regularizer depends on the neighborhood. This distinction is central to interpreting reported guarantees and mechanisms.

2. Continuous-state nearest-neighbor Q-learning

In continuous-state discounted MDPs with unknown transition kernels, Shah and Xie formulate Q-learning from a single dependent trajectory by combining Bellman updates with nearest-neighbor regression (Shah et al., 2018). The setting is an infinite-horizon discounted MDP $(X,A,p,r,\gamma)$ with compact continuous state space $X \subset \mathbb{R}^d$, finite action set $A$, Lipschitz reward and transition kernel, and only one sample path under an arbitrary policy. The optimal Q-function satisfies

$$Q^*(x,a)=r(x,a)+\gamma\,\mathbb{E}_{x'\sim p(\cdot\mid x,a)}\!\left[\max_{a'\in A}Q^*(x',a')\right],$$

and, under the stated regularity assumptions, is bounded and Lipschitz.

The 2018 construction discretizes the state space by an $h$-net $k$0 and extends grid values to the continuous space by a nearest-neighbor operator

$k$1

where $k$2 is nonnegative, sums to one, and has support limited by radius $k$3 (Shah et al., 2018). This yields the joint Bellman-nearest-neighbor operator

$k$4

which is a $k$5-contraction on bounded functions over $k$6. The algorithm alternates inner incremental averaging updates along the observed trajectory with outer updates performed once all ball-action pairs have been visited.

The finite-sample theorem is stated in sup norm. With $k$7, $k$8, $k$9, and bandwidth choice $Q$0, the method returns an estimator $Q$1 satisfying $Q$2 with probability at least $Q$3 after

$Q$4

samples, where $Q$5 is the expected cover time at granularity $Q$6 (Shah et al., 2018). Under the “well-behaved MDP” example in the paper, $Q$7 scales as $Q$8 under a random or $Q$9-greedy policy, giving overall complexity $\kappa$0. The same work also proves a lower bound of $\kappa$1, making the curse of dimensionality explicit.

Zhao and Lai replace grid-based discretization with direct $\kappa$2-NN smoothing and show that more efficient sample reuse improves the $\kappa$3-dependence (Zhao et al., 2023). Their offline method keeps all samples and iterates

$\kappa$4

while the online variant uses a recency buffer $\kappa$5 and the schedule

$\kappa$6

For bounded state spaces, the paper gives high-probability sup-norm rates corresponding to

$\kappa$7

offline and

$\kappa$8

online, and argues that the $\kappa$9-dependence is minimax-optimal (Zhao et al., 2023). A further difference is scope: the 2023 formulation explicitly treats possibly unbounded state spaces under tail and mixing assumptions.

3. Neighborhood-conditioned Q-learning in graph-structured multi-agent systems

In graph-based MARL, neighbor-specificity often appears not as geometric smoothing but as graph-local representation learning. In Neighborhood Cognition Consistent MARL, the environment is a graph $Q_{i,j}(s,a)$0 with neighbor set $Q_{i,j}(s,a)$1 for each agent, and a shared-parameter GCN aggregates neighborhood features by

$Q_{i,j}(s,a)$2

The resulting cognition is decomposed into an agent-specific component $Q_{i,j}(s,a)$3 and a neighborhood-specific latent variable $Q_{i,j}(s,a)$4, and the per-agent value is mixed as

$Q_{i,j}(s,a)$5

Neighbor-specificity is enforced by a cognitive-dissonance term that averages $Q_{i,j}(s,a)$6 only over $Q_{i,j}(s,a)$7 rather than over all agents (Mao et al., 2019). Here locality acts as a regularizer on latent cognition rather than as a per-neighbor Q-table.

A different variant uses the neighborhood only through a coarse environmental state. In the spatial public-goods game, agents on a square lattice observe the number of cooperating and defecting neighbors and map the local environment into three states: $Q_{i,j}(s,a)$8 if $Q_{i,j}(s,a)$9, $(X,A,p,r,\gamma)$0 if $(X,A,p,r,\gamma)$1, and $(X,A,p,r,\gamma)$2 if $(X,A,p,r,\gamma)$3 (Zheng et al., 2024). Each agent then applies

$(X,A,p,r,\gamma)$4

with action set $(X,A,p,r,\gamma)$5 in the compulsory public-goods game and $(X,A,p,r,\gamma)$6 in the voluntary version. The reported thresholds are $(X,A,p,r,\gamma)$7 for the Fermi rule, $(X,A,p,r,\gamma)$8 for Q-learning in the compulsory game, and $(X,A,p,r,\gamma)$9 for Q-learning with voluntary participation (Zheng et al., 2024). The paper emphasizes that cooperation under Q-learning does not rely on the stable cooperator clusters characteristic of imitation dynamics.

Neighbor-conditioned reward shaping constitutes another usage. In multi-agent traffic signal control, each intersection agent updates its local Q-value with its own fuzzy reward plus a weighted sum of neighbor rewards,

$X \subset \mathbb{R}^d$0

where $X \subset \mathbb{R}^d$1 is obtained by a weighting FIS using local traffic conditions and green times (Daeichian et al., 2019). In the reported five-intersection simulation, total average delay decreases from more than $X \subset \mathbb{R}^d$2 s under fixed-time scheduling to approximately $X \subset \mathbb{R}^d$3 s under the proposed neighbor-aware fuzzy Q-learning.

The structured-cooperation study based on reputation-shaped reinforcement shows a looser use of the term (Du et al., 29 Jan 2026). There, Q-values are not maintained per neighbor; instead, each agent has a $X \subset \mathbb{R}^d$4 table over its own previous action and current action,

$X \subset \mathbb{R}^d$5

with combined reinforcement

$X \subset \mathbb{R}^d$6

The paper explicitly notes that the implementation is not strict per-neighbor Q-learning: neighbor effects arise through local payoff aggregation and repeated interactions on the lattice rather than through neighbor-indexed Q-values. This usage is important because it prevents an overly narrow identification of neighbor-specificity with per-edge storage.

4. Explicit per-neighbor learning: partner selection, pairwise values, and rewiring

A stronger notion of neighbor-specificity appears when agents learn separate values for distinct neighbors or separate policies for interaction selection. In the spatial Prisoner’s Dilemma framework with selective interaction, each agent is equipped with two independent Q-networks: an interaction-selection network $X \subset \mathbb{R}^d$7 over $X \subset \mathbb{R}^d$8 and a dilemma-strategy network $X \subset \mathbb{R}^d$9 over $A$0 (Ren et al., 2024). The neighbor-selection state is $A$1 and records, over the previous $A$2 timesteps, each neighbor’s dilemma action together with reciprocal selection flags. The dilemma-strategy state is $A$3 and stores the recent one-hot dilemma actions of the focal agent and its four neighbors. Both networks are trained from the same scalar utility $A$4 through standard DQN losses with target networks and prioritized replay.

This formulation makes partner management a learned control problem. Actual interactions occur only under reciprocal selection, with realized degree

$A$5

and raw payoff

$A$6

Long-term experience is introduced through a weighted moving-average payoff and memory length $A$7, with $A$8 (Ren et al., 2024). The reported dynamics show strategic assortment: early episodes have $A$9 CC links versus $$Q^*(x,a)=r(x,a)+\gamma\,\mathbb{E}_{x'\sim p(\cdot\mid x,a)}\!\left[\max_{a'\in A}Q^*(x',a')\right],$$0 DD links, and the framework maintains full cooperation until the dilemma strength exceeds $$Q^*(x,a)=r(x,a)+\gamma\,\mathbb{E}_{x'\sim p(\cdot\mid x,a)}\!\left[\max_{a'\in A}Q^*(x',a')\right],$$1. The same study reports that with four-step memory ($$Q^*(x,a)=r(x,a)+\gamma\,\mathbb{E}_{x'\sim p(\cdot\mid x,a)}\!\left[\max_{a'\in A}Q^*(x',a')\right],$$2), cooperation is sustained when $$Q^*(x,a)=r(x,a)+\gamma\,\mathbb{E}_{x'\sim p(\cdot\mid x,a)}\!\left[\max_{a'\in A}Q^*(x',a')\right],$$3 rises from $$Q^*(x,a)=r(x,a)+\gamma\,\mathbb{E}_{x'\sim p(\cdot\mid x,a)}\!\left[\max_{a'\in A}Q^*(x',a')\right],$$4 to $$Q^*(x,a)=r(x,a)+\gamma\,\mathbb{E}_{x'\sim p(\cdot\mid x,a)}\!\left[\max_{a'\in A}Q^*(x',a')\right],$$5, whereas with single-step memory ($$Q^*(x,a)=r(x,a)+\gamma\,\mathbb{E}_{x'\sim p(\cdot\mid x,a)}\!\left[\max_{a'\in A}Q^*(x',a')\right],$$6) cooperation drops from $$Q^*(x,a)=r(x,a)+\gamma\,\mathbb{E}_{x'\sim p(\cdot\mid x,a)}\!\left[\max_{a'\in A}Q^*(x',a')\right],$$7 to $$Q^*(x,a)=r(x,a)+\gamma\,\mathbb{E}_{x'\sim p(\cdot\mid x,a)}\!\left[\max_{a'\in A}Q^*(x',a')\right],$$8.

An even more explicit edge-local scheme is given by Q-learning-driven adaptive rewiring on scale-free networks (Weng et al., 1 Sep 2025). For each edge $$Q^*(x,a)=r(x,a)+\gamma\,\mathbb{E}_{x'\sim p(\cdot\mid x,a)}\!\left[\max_{a'\in A}Q^*(x',a')\right],$$9, the algorithm maintains two Q-functions: $h$0 for cooperation or defection and $h$1 for rewiring decisions. The local state is a three-valued indicator, $h$2 if both defected, $h$3 if one cooperated and one defected, and $h$4 if both cooperated. The two TD updates are

$h$5

and

$h$6

where $h$7 aggregates rewards over the rewiring timescale (Weng et al., 1 Sep 2025).

Because every relationship has its own action and rewiring values, the method learns both whom to cooperate with and whom to drop. The paper reports three regimes as the rewiring constraint $h$8 varies: a permissive regime at low $h$9, an intermediate regime with sensitive dependence on dilemma strength, and a patient regime at high $k$00 (Weng et al., 1 Sep 2025). At $k$01, cooperation exceeds $k$02 for $k$03 up to approximately $k$04, and simulations scale to networks with $k$05. This is one of the clearest cases in which “neighbor-specific Q-learning” means literal per-neighbor credit assignment.

5. Local-information critics and distributed control

In networked linear-quadratic control, neighbor-specific Q-learning takes a control-theoretic rather than game-theoretic form. For systems with decoupled costs and spatially exponentially decaying dynamics, the networked LQR study shows that each agent’s individual value and Q-functions inherit spatial decay away from the agent (Olsson et al., 2024). Under a fixed linear feedback $k$06, the individual value is

$k$07

where

$k$08

and the individual Q-function is

$k$09

The paper proves that the blocks of $k$10 are spatially exponentially decaying away from agent $k$11, and that the $k$12-truncation error satisfies

$k$13

This justifies localized LSTDQ critics that use only $k$14-neighborhood states and actions (Olsson et al., 2024).

The resulting actor-critic algorithm estimates truncated individual Q-functions from local features and updates a $k$15-sparse controller. Critic features are built from $k$16, and overlap between local estimates reconstructs the sparse global critic. In the reported $k$17 thermal-network simulation, spatial decay is observed empirically, and learned local critics produce near-optimal performance relative to centralized optimal control (Olsson et al., 2024).

A second distributed-control meaning of neighbor-specificity appears in MPC-based distributed Q-learning (Mallick et al., 20 Nov 2025). There, the Q-function is represented by a distributed convex MPC program

$k$18

subject to local constraints and coupled dynamics involving neighbors. Sensitivity analysis gives

$k$19

and the structured parameterization implies that mixed second derivatives vanish across agents, yielding block-diagonal Hessian terms (Mallick et al., 20 Nov 2025). The distributed second-order update takes the exact local form

$k$20

Only local curvature information and small consensus-reduced scalar aggregates are required. In the three-agent chain experiment, the second-order distributed method uses $k$21 versus $k$22 for the first-order distributed method and is reported to reduce TD error and global stage cost substantially faster (Mallick et al., 20 Nov 2025).

6. Theoretical themes, misconceptions, and open directions

Across these formulations, a first recurring theme is that locality trades statistical or computational scalability against approximation error. In continuous-state nearest-neighbor methods, smaller bandwidths or larger neighborhood resolution reduce bias but increase covering numbers, cover times, and sensitivity to dimension; the 2018 and 2023 analyses both make the curse of dimensionality explicit, and the 2023 paper identifies $k$23 as minimax-optimal in $k$24 up to logarithmic factors (Shah et al., 2018, Zhao et al., 2023). In networked control, increasing $k$25 decreases structural truncation error exponentially but enlarges local feature dimension, communication range, and estimation variance (Olsson et al., 2024).

A second theme is that “neighbor-specific” does not imply a single algorithmic template. Some methods are strictly pair-local, with one Q-object per edge or per partner (Weng et al., 1 Sep 2025). Others are neighborhood-conditioned, using local summaries such as counts of cooperating neighbors, GCN-aggregated latent representations, or neighbor-weighted rewards (Mao et al., 2019, Zheng et al., 2024). The reputation-shaped lattice model is explicit that its implementation does not maintain Q-values per neighbor at all, even though local neighbor effects are central to the dynamics (Du et al., 29 Jan 2026). A common misconception is therefore to equate the term exclusively with per-neighbor Q-tables.

A third theme concerns horizon and exploration. In nearest-neighbor continuous-state RL, convergence depends on coverage or on full-support behavior policies, and near-undiscounted regimes are statistically harder (Shah et al., 2018, Zhao et al., 2023). In cooperation studies, discounting changes the qualitative effect of local information: the reputation-shaped work reports that the promoting effect of reputation vanishes as $k$26, while the public-goods study finds a non-monotonic dependence on $k$27 inside a voluntary-participation regime (Du et al., 29 Jan 2026, Zheng et al., 2024). These results suggest that neighborhood information is not beneficial independently of temporal credit assignment.

Finally, the status of guarantees remains uneven. Continuous-state nearest-neighbor methods and control-theoretic local critics come with finite-sample bounds, lower bounds, or structural approximation theorems (Shah et al., 2018, Zhao et al., 2023, Olsson et al., 2024). By contrast, several graph-based MARL formulations emphasize empirical performance and explicitly note the absence of formal convergence proofs under shaped rewards, variational regularization, or dual-network partner selection (Mao et al., 2019, Ren et al., 2024, Du et al., 29 Jan 2026). Distributed second-order control methods additionally assume convex MPC subproblems, synchronous communication, and connected graphs (Mallick et al., 20 Nov 2025). Open directions appearing across the literature include continuous action spaces, adaptive bandwidth or variable-radius neighborhoods, dynamic topologies, off-policy correction, richer local function approximators, explicit communication constraints, and analytical treatment of non-stationary local interaction dynamics (Zhao et al., 2023, Weng et al., 1 Sep 2025, Mallick et al., 20 Nov 2025).

In that broader sense, Neighbor-Specific Q-Learning is best understood not as a single algorithm but as a locality principle for Q-learning. It encompasses nonparametric smoothing in continuous state spaces, graph-local representation learning, explicit per-edge partner management, and distributed control with neighborhood-constrained critics and updates. The differences among these formulations are substantial, but they are unified by the same structural premise: value estimation becomes tractable, communicable, or strategically meaningful when organized around neighborhoods rather than around fully global state-action descriptions.