---
title: Neighborhood Attention Decoder (NAD)
url: https://www.emergentmind.com/topics/neighborhood-attention-decoder-nad
type: topic
---

# Neighborhood Attention Decoder (NAD)

Neighborhood Attention Decoder (NAD) most explicitly denotes the decoder introduced in SurGe, where a ViT token grid is progressively upsampled and mixed with Neighborhood Attention to improve local surface geometry in dense monocular point-map prediction [2605.31577]. The phrase is not standardized across the literature. Closely related works use the distinct names “Neighbor-Aware Decoder” in point-of-interest recommendation [1809.10770] and “Neighborhood decoder” in single-view 3D reconstruction [2307.09112], while other papers define neighborhood-based attention operators without introducing a decoder under that name [2204.07143], [2209.15001]. This suggests that NAD is best treated as a precise architectural term in SurGe and as a broader descriptive label only with care.

## 1. Terminological scope and nomenclature

The literature surrounding NAD is heterogeneous. Several papers are relevant because they use neighborhood-restricted aggregation in a decoder or decoder-like role, but they do not refer to the same object.

| Work | Exact term | Architectural role |
|---|---|---|
| SurGe [2605.31577] | Neighborhood Attention Decoder | Five-stage progressive decoder for point maps |
| SAE-NAD [1809.10770] | Neighbor-Aware Decoder | Additive geographic influence term in an autoencoder decoder |
| NU-MCC [2307.09112] | Neighborhood decoder | Local query-point decoder around predicted anchors |
| NAT [2204.07143] | Neighborhood Attention | Attention primitive and hierarchical backbone |
| MNA-GT [2211.07970] | Multi-Neighborhood Attention based Graph Transformer | Encoder-style graph Transformer, not a decoder |

This distinction matters because a recurrent misconception is to collapse all of these uses into a single decoder family. In SAE-NAD, “NAD” expands to **Neighbor-Aware Decoder**, not Neighborhood Attention Decoder, and the decoder does not use query-key-value attention; it adds a geographically weighted influence term to reconstruction scores [1809.10770]. In NU-MCC, the exact name is **Neighborhood decoder**, and its local aggregation is attached to a 3D implicit reconstruction pipeline rather than to a dense image decoder [2307.09112]. In MNA-GT, neighborhood information is encoded through multiple attention kernels inside a graph Transformer encoder, and the paper explicitly does not define a decoder module in the encoder–decoder sense [2211.07970].

## 2. Canonical formulation in SurGe

In the most direct use of the term, SurGe places NAD between a DINOv2-initialized ViT-Large encoder and a point-map output head [2605.31577]. For an input image $\mathbf{I}\in\mathbb{R}^{H\times W\times 3}$, the encoder produces

$$
\mathbf{Z}\in\mathbb{R}^{\frac{H}{16}\times \frac{W}{16}\times C},
$$

and NAD upsamples and decodes these features into a full-resolution point map $\hat{\mathbf{P}}$.

The decoder consists of five stages. Stage $\ell$ operates on

$$
\mathbf{X}_\ell \in \mathbb{R}^{\frac{H}{2^{5-\ell}} \times \frac{W}{2^{5-\ell}} \times C_\ell},
$$

with channel schedule

$$
(C_1,\ldots,C_5)=(1024,512,256,128,64),
$$

and with

$$
n_1=\cdots=n_5=3.
$$

Stages $1$ through $4$ upsample by a factor of $2\times$; for $\ell<5$, the upsampling module is a transposed $2\times2$ convolution with stride $2$, followed by a $3\times3$ convolution. The paper does not describe U-Net-style lateral skip connections from intermediate backbone stages; the decoder is attached to the final encoder token grid rather than to a multi-level skip-fusion pathway [2605.31577].

A mathematically faithful reconstruction of the stage computation is

$$
\mathbf{X}_\ell^{(0)} = \mathbf{X}_\ell + \mathbf{E}_\ell^{UV},
$$

$$
\mathbf{Y}_\ell^{(b)} = \mathbf{X}_\ell^{(b-1)} + \mathrm{NA}_\ell^{(b)}(\mathbf{X}_\ell^{(b-1)}),
$$

$$
\mathbf{X}_\ell^{(b)} = \mathbf{Y}_\ell^{(b)} + \mathrm{FFN}_\ell^{(b)}(\mathbf{Y}_\ell^{(b)}),
$$

followed, for $\ell<5$, by

$$
\mathbf{X}_{\ell+1} = \mathrm{Conv}_{3\times3}\!\left(\mathrm{Deconv}_{2\times2,s=2}\!\left(\mathbf{X}_\ell^{(n_\ell)}\right)\right).
$$

At the final stage, the output head predicts $(\xi,\eta,\rho)$ and maps them to 3D points as

$$
(\xi e^\rho,\eta e^\rho,e^\rho).
$$

Thus $z=e^\rho$, and the final point map is obtained by projecting decoder features into a point-map parameterization rather than by direct Cartesian regression [2605.31577].

Several implementation details are integral to the definition of NAD in this paper. Each NAD block uses Neighborhood Attention with attention window $k=9$, head dimension $d_h=64$, and FFN hidden dimension $4C_\ell$. Queries and keys use window-matched RoPE, each stage adds a learned UV positional embedding, the blocks use QK normalization, and the architecture removes the usual pre-attention and pre-FFN LayerNorm [2605.31577].

## 3. Neighborhood Attention as the operative mechanism

NAD inherits its attention operator from Neighborhood Attention as formalized in NAT [2204.07143]. For input $X \in \mathbb{R}^{n \times d}$ with linear projections $Q$, $K$, and $V$, and relative positional biases $B(i,j)$, the attention logits for token $i$ over a local neighborhood are defined by the nearest spatial neighbors of that query token:

$$
\mathbf{A}_{i}^{k} =
\begin{bmatrix}
Q_i K_{\rho_{1}(i)}^T + B_{(i,\rho_{1}(i))} \\
Q_i K_{\rho_{2}(i)}^T + B_{(i,\rho_{2}(i))} \\
\vdots \\
Q_i K_{\rho_{k}(i)}^T + B_{(i,\rho_{k}(i))}
\end{bmatrix},
$$

with value matrix

$$
\mathbf{V}_{i}^{k} =
\begin{bmatrix}
V_{\rho_{1}(i)}^T & V_{\rho_{2}(i)}^T & \hdots & V_{\rho_{k}(i)}^T
\end{bmatrix}^{T},
$$

and output

$$
\mathrm{NA}_{k}(i) = \mathrm{softmax}\left(\frac{\mathbf{A}_{i}^{k}}{\sqrt{d}}\right)\mathbf{V}_{i}^{k}.
$$

The crucial structural point is that Neighborhood Attention is a **sliding-window** attention pattern rather than a blocked window partition. Each query token attends only to its nearest neighbors, but adjacent queries have overlapping neighborhoods. NAT characterizes this as a pixel-wise operation with linear time and space complexity relative to image size, in contrast to the quadratic complexity of full self-attention [2204.07143]. SurGe uses this operator as the core mixing mechanism of a progressive decoder rather than as a backbone-only primitive [2605.31577].

The broader neighborhood-attention design space is larger than SurGe’s specific choice. DiNAT introduces Dilated Neighborhood Attention, which expands receptive fields at no additional cost and alternates NA and DiNA inside a hierarchical vision transformer [2209.15001]. Generalized Neighborhood Attention further unifies sliding-window attention, strided sliding-window attention, and blocked attention through a stride parameter, and it explicitly discusses causal masking as compatible with the NA family [2504.16922]. These papers do not define an NAD architecture, but they establish the operator lineage from which decoder variants can be built. A plausible implication is that SurGe’s NAD occupies the “plain local self-attention” point in this larger family: progressive, multiscale, and locality-biased rather than globally attentive or dilation-based.

## 4. Other decoder lineages with NAD-adjacent behavior

The nearest earlier use of the acronym is SAE-NAD, a self-attentive autoencoder for point-of-interest recommendation in which NAD stands for **Neighbor-Aware Decoder** [1809.10770]. The decoder augments the standard autoencoder reconstruction with a POI-neighborhood term derived from embedding similarity and a geographic RBF kernel. If $L_u$ is the visited-POI set for user $u$, the neighbor-aware influence matrix is

$$
\mathbf{P}_{u} = (\mathbf{W}^{(4)} \cdot \mathbf{W}^{(1)}[L_{u}]) \odot \mathbf{K}[L_{u}],
$$

which is then summed over visited POIs to form $\mathbf{p}_u$, and the final reconstruction becomes

$$
\hat{\mathbf{x}_{u}} = a_{4}(\mathbf{W}^{(4)} \mathbf{z}_{u}^{(3)} + \mathbf{p}_{u} + \mathbf{b}^{(4)}).
$$

The paper explicitly notes that this decoder is not attention in the strict neural-attention sense: there is no query-key-value computation and no softmax over decoder neighbors [1809.10770]. Its relevance to NAD is therefore conceptual rather than terminological: it is a decoder-side neighborhood mechanism, but not a Neighborhood Attention Decoder.

NU-MCC provides a different lineage. Its **Neighborhood decoder** addresses single-view 3D reconstruction by introducing predicted anchors, described in the abstract as “center points,” as a sparse proxy for input visual features [2307.09112]. The anchor representation is

$$
\mathbf{F}_c = \{\mathbf{Z}_c, \mathbf{X}_c\} = \mathcal{G}(\mathbf{R}),
$$

where $\mathbf{Z}_c$ are anchor features and $\mathbf{X}_c$ are anchor positions. For a query point $\mathbf{q}$, the decoder aggregates only the $m$-nearest anchors:

$$
\mathbf{z}^*_{\mathbf{q}} = \sum_{i=1}^{m} \mathbf{W}^i \odot \left(\mathbf{Z}_{c,\mathbf{q}}^i W_v\right),
$$

with weights

$$
\mathbf{W} = \sigma\!\left( \psi\!\left( \mathbf{z}W_q + \mathbf{Z}_{c,\mathbf{q}}W_k + \delta(\Delta_{\mathbf{q}}) \right) \right).
$$

The per-query predictions are then $f(\mathbf{q}), c(\mathbf{q}) = \Psi(\mathbf{q}, \mathbf{z}^{*}_{\mathbf{q}})$. This decoder is neighborhood-restricted and attention-like, but the exact paper term remains “Neighborhood decoder,” not NAD [2307.09112].

A third decoder-adjacent route appears in LiDAR geometry compression through Neighborhood Point Attention and NPAFormer [2208.12573]. For sparse tensor inputs with coordinates $C_{in}$ and features $F_{in}$, the local neighborhood is built by coordinate-space $k$-NN, and relative geometry is encoded as

$$
F_{e}=concat(F_{\rm kNN}, C_{\rm kNN}-C_{in}).
$$

Queries, keys, and values are then

$$
Q=F_{in} \cdot W_{Q}, \qquad K_{\rm kNN} = F_{e} \cdot W_{K}, \qquad V_{\rm kNN} = F_{e} \cdot W_{V},
$$

and localized attention is

$$
{\rm NPA}(Q,K_{\rm kNN},V_{\rm kNN}) = softmax\!\left(\frac{QK_{\rm kNN}^{T}}{\sqrt{d_e}}\right)V_{\rm kNN}.
$$

This module is embedded in a multiscale occupancy probability approximation used directly in arithmetic decoding. It is not named NAD, but it is decoder-side neighborhood attention in a literal probabilistic decoding loop [2208.12573].

## 5. Empirical profile

SurGe motivates NAD by arguing that recent feedforward point-map models achieve strong global geometry while still exhibiting poor local surface geometry, particularly on thin structures and high-frequency detail [2605.31577]. In the decoder ablation, NAD outperforms ViT decoder, DPT head, ConvStack, and ConvStack-L on the reported local point metric across all five listed datasets. On ETH3D, for example, NAD reaches $2.66$ versus $2.94$ for ConvStack-L and $4.12$ for the ViT decoder; on DIODE it reaches $4.63$ versus $4.82$ and $6.25$, respectively. On the point-map normal angular error, NAD is again best on all reported datasets, including $18.3$ on ETH3D, $16.5$ on iBims-1, $10.5$ on GSO, $24.5$ on Sintel, and $12.0$ on DIODE. The same study reports that NAD also attains the best average rank, $1.31$, on the global point-map metric, with first place on $7$ of $8$ datasets. These gains are not free: at $512\times512$, decoder-only runtime is $13.56$ ms for NAD versus $5.94$ ms for ConvStack-L, and the full model with DINOv2-Large is $21.15$ ms versus $14.53$ ms, with peak memory rising from $1.87$ GiB to $2.02$ GiB [2605.31577].

NU-MCC offers a complementary efficiency-oriented result for local neighborhood decoding in 3D reconstruction [2307.09112]. The paper reports that NU-MCC “outperforms MCC by 9.7% in terms of the F1-score on the CO3D-v2 dataset with more than 5x faster running speed.” Table 1 isolates the decoder contribution: MCC with occupancy takes $18.5$ s at evaluation with $216$k query points, whereas the neighborhood decoder without fine features takes $1.2$ s and with fine features $1.5$ s. This supports the claim that local query-restricted decoding can reduce the cost of large query sets without reverting to global Transformer decoding [2307.09112].

SAE-NAD provides evidence from recommendation rather than geometry [1809.10770]. In its component study, the neighbor-aware decoder alone outperforms the weighted autoencoder and also exceeds the self-attentive encoder-only variant on all three datasets and all listed ranking metrics. On Gowalla at P@10 / R@10 / MAP@10, WAE scores $0.05599 / 0.13819 / 0.06728$, SAE-WAE scores $0.06039 / 0.14808 / 0.07257$, and NAD-WAE scores $0.07029 / 0.17915 / 0.08699$. The paper interprets this as evidence that decoder-side neighborhood modeling is especially effective because it directly affects final recommendation scores [1809.10770].

In LiDAR compression, neighborhood attention also shows a decoder-facing systems benefit [2208.12573]. Relative to OctAttention, the reported average decoding time drops from $3529.72$ s to $5.55$ s, a speedup of about $640\times$, while still presenting better compression efficiency. Relative to SparsePCGC, decoding time drops from $11.77$ s to $5.55$ s, about $2\times$ faster. The paper further reports $>14\%$ bitrate reduction for the lossless scenario and about $17.7\%$ BD-rate gain over G-PCC in lossy coding [2208.12573].

## 6. Architectural significance and open distinctions

Across these works, the phrase “Neighborhood Attention Decoder” refers to structurally different objects. In SurGe, NAD is a multistage dense decoder that uses plain Neighborhood Attention for local content-dependent mixing over 2D feature maps [2605.31577]. In NU-MCC, the decoder is neighborhood-restricted because each query point attends only to nearby anchor and fine features in continuous 3D space [2307.09112]. In SAE-NAD, the decoder is neighborhood-aware because it injects an explicit POI-to-POI influence field based on embedding similarity and geographic proximity, not because it implements self-attention [1809.10770]. In LiDAR compression, neighborhood attention lives inside a conditional probability model used in arithmetic decoding over sparse occupied voxels [2208.12573].

This diversity exposes three stable axes of variation. The first is **how the neighborhood is defined**: fixed local image windows in NA-based dense decoders, nearest anchors in implicit 3D reconstruction, coordinate-space $k$-NN in sparse geometry, or geography-weighted item neighborhoods in recommendation. The second is **what role the decoder plays**: dense feature upsampling, continuous query evaluation, score reconstruction, or probabilistic entropy decoding. The third is **how locality trades against efficiency**. SurGe explicitly treats NAD as an accuracy-oriented decoder and states that more efficient variants are a useful future direction [2605.31577]. Generalized Neighborhood Attention and NABLA extend the operator space further through stride-controlled sparse patterns and adaptive block-level masks, but they do not define a decoder architecture called NAD [2504.16922], [2507.13546].

A plausible implication is that NAD should not be treated as a single canonical module. The strongest common denominator is decoder-side computation that restricts or reweights interactions according to a neighborhood structure rather than relying on unrestricted global attention or purely fixed-kernel convolution. Under that reading, SurGe provides the clearest explicit NAD, while the surrounding literature supplies adjacent mechanisms, competing nomenclatures, and alternative definitions of what “neighborhood” means in a decoder.

Source: https://www.emergentmind.com/topics/neighborhood-attention-decoder-nad