---
title: Neighboring Autoregressive Modeling (NAR)
url: https://www.emergentmind.com/topics/neighboring-autoregressive-modeling-nar
type: topic
---

# Neighboring Autoregressive Modeling (NAR)

Neighboring Autoregressive Modeling (NAR) encompasses a set of paradigms for sequence modeling and generation in diverse modalities—text, vision, speech, and networked systems—where the autoregressive dependencies are restructured to exploit locality, parallelism, and explicit neighbor relations. In contrast to standard left-to-right (chain rule) autoregressive models and fully independent (non-autoregressive) approaches, NAR leverages the structural or spatial-temporal neighborhood of tokens or nodes to improve efficiency, robustness, and sometimes output quality.

## 1. Conceptual Foundations and Probabilistic Factorization

Neighboring Autoregressive Modeling (NAR) operates at the interface between two extremes: fully autoregressive (AR) models, which factorize the joint distribution as $p_{AR}(x_1,\ldots,x_n) = \prod_{i=1}^n p(x_i|x_{<i})$, and non-autoregressive (NAR) models, which factorize as $p_{NAR}(x_1,\ldots,x_n)=\prod_{i=1}^n p(x_i)$ or, more generally, as conditionally independent given the input (e.g., $p(y_t|x)$ in sequence-to-sequence tasks) [2509.20744][2004.10454][2204.09269]. NAR introduces dependency structures based on proximity or explicit network relations, restructuring the decomposition to favor prediction of tokens/nodes based on their spatial, temporal, or structural neighbors.

For example, in visual generation, NAR factorization is performed by grouping tokens according to ascending Manhattan distance from a seed, so that all tokens at distance $d$ are predicted in parallel, conditioned on those at distances $<d$ [2503.10696]. In networked time series, NAR parameterizes the evolution of node $i$ as a function of its own past and pasts of its explicit neighbors, with flexible weighting [2110.09596][2007.05521].

## 2. Visual NAR: Locality-Preserving Outpainting

In visual autoregressive modeling, vanilla approaches flatten images or videos into 1D sequences, applying next-token prediction left-to-right, top-to-bottom, ignoring the fundamental spatial locality. Neighboring Autoregressive Modeling (NAR) for vision imposes an outpainting process: generation progresses from a seed token, expanding in shells of constant spatial (or spatiotemporal) Manhattan distance [2503.10696]. At each step $d$, the set $S_d$ of tokens at distance $d$ from the seed are generated in parallel, conditioned only on tokens at $S_{<d}$.

This factorization preserves exact autoregressive semantics:

\[
p(\{x\}) = p(S_0) \prod_{d=1}^{D_{max}} p(S_d | S_{<d})
\]

where $S_d$ are token sets equidistant from the initial seed. NAR's inference complexity reduces from $O(n^2)$ (for $n\times n$ images) to $O(n)$ forward passes. This is achieved via dimension-oriented decoding heads (horizontal, vertical, temporal), each responsible for distinct directions of outgrowth. When a token can be predicted from multiple directions, the outputs are ensembled by mixing logits. Empirical results on ImageNet-256 and UCF-101 show NAR surpasses both raster-order AR and parallel block-based auto-regressors (PAR-4X) in throughput (2.4–8.6×) and in FID/FVD scores [2503.10696].

## 3. NAR in Structured Sequence and Network Models

In networked dynamic systems, the term "Neighboring Autoregressive" or "Network Autoregressive" (network NAR) refers to models where each node's value depends both on its own historical states and those of directly connected nodes [2110.09596][2402.07373][2007.05521]. For node $i$ at time $t$:

\[
X_{i,t} = \sum_{j=1}^{q_1} a_i^{(j)} X_{i,t-j} + \sum_{j=1}^{q_2} b_i^{(j)} \sum_{k=1}^N w_{ik} X_{k, t-j} + Z_{i,t}^\top\gamma_i + \varepsilon_{i,t},
\]

where $w_{ik}$ are elements of a normalized adjacency matrix. This approach generalizes classical vector autoregression (VAR) by encoding explicit neighbor relations, facilitating the modeling of spatial/temporal spillovers and heterogeneity.

Extensions include Community NAR (CNAR), wherein block structures capture intra- and inter-community effects [2007.05521], and Functional Coefficient NAR (FCNAR), allowing coefficients to vary nonlinearly with regime variables [2402.07373]. Stationarity is analyzed via spectral radius conditions on companion matrices; estimation frameworks include OLS, GLS, ridge, and two-stage weighted methods, with rigorous theoretical guarantees [2110.09596][2007.05521].

## 4. NAR for Efficient Language Model Reasoning

In cognitive and language tasks requiring multi-step reasoning, a hybrid AR–NAR framework leverages the benefits of both paradigms. Specifically, "Parallel Thinking, Sequential Answering" [2509.20744] decouples the high-level plan (reasoning trace) from the final answer surface realization. The system operates as follows:

1. A Discrete Diffusion Language Model (the NAR "Mercury Coder") generates an explicit intermediate trace $z_0$ in a sequence of $T$ parallel denoising steps:

   \[
   z_{t-1}^{(i)} \sim p_\theta(z_{t-1}^{(i)}|z_t, x)
   \]

   for all positions $i$ in parallel at each diffusion step.

2. This plan $z_0$ is prepended to the problem prompt and provided to a powerful AR decoder (e.g., GPT-5), which then produces the final output $y$ token by token, sequentially maximizing $p_\varphi(y|x, z_0)$.

The NAR-generated reasoning trace mitigates long-horizon planning errors by global parallel refinement, while the AR stage focuses on fine-grained correctness and fluency. This division yields a 26 percentage point lift in pass@1 on combined reasoning benchmarks and a 30–40% wall-clock reduction in generation time compared to pure AR [2509.20744].

## 5. NAR in Speech and Text: Parallel Sequence Generation

Neighboring and non-autoregressive paradigms dominate recent work in efficient sequence generation beyond vision and networks, notably in speech recognition and neural machine translation (NMT). Here, NAR factorization assumes conditional independence among (output) tokens given the input. Canonical approaches include:

- CTC-based models (Connectionist Temporal Classification), which align input frames to target tokens under monotonicity, enabling parallel decoding [2110.05249][2206.08317].
- Masked language model refinement (Mask-CTC, CMLM), where tokens are initially hypothesized in parallel (possibly with masks for uncertain positions), and filled via successive passes [2110.05249][2207.10600].
- Iterative edit-based models (Levenshtein Transformer, NeighborEdit), where output hypotheses are refined in a small number of parallelized edit operations, often leveraging nearest-neighbor initialization in the latent space to guide NAR decoders and reduce iteration count without sacrificing output quality [2208.12496].

NAR's critical challenge is modeling target-side dependencies—substitution and deletion errors are common when dependencies are ignored. Techniques to address this include glancing language model samplers, knowledge distillation from AR teachers, and auxiliary alignment constraints. For instance, the Paraformer architecture introduces a continuous integrate-and-fire predictor for length estimation and a GLM sampler for partial conditioning, attaining AR-level error rates at up to 12× speedup [2206.08317]. Mask-CTC with AR-to-NAR knowledge distillation further reduces the gap, with sub-10× model size and moderate ∼1% absolute WER/CER loss [2207.10600].

## 6. Theoretical and Practical Properties, Limitations

NAR methods offer massive decoding speedup—single- or few-pass parallel decoders can outperform AR models in throughput by one or more orders of magnitude [2503.10696][2206.08317][2204.09269]. However, the independence assumption can degrade output quality when output tokens exhibit strong interdependence, as quantified by the attention density ratio $R(1)$ in target-masked "CoMMA" analysis [2004.10454]. In tasks with high token dependency (e.g., ASR), pure NAR models still lag AR models in accuracy unless supplemented with distillation, alignment, or auxiliary AR rescoring.

Structural NAR models (e.g., network time series) hinge on accurate specification of neighbor relations and careful control of stability (spectral radius). Empirically, these models yield improved predictive performance when local dependencies are strong and community or covariate heterogeneity is present [2110.09596][2007.05521].

In vision, locality-preserving NAR achieves near-optimal tradeoffs between efficiency and perceptual quality for image/video generation, whereas naive flattening-based AR or block-based NAR remains suboptimal [2503.10696].

## 7. Perspectives and Research Directions

Neighboring Autoregressive Modeling, broadly construed, continues to drive advances in high-efficiency sequence generation by bridging locality, dependency, and parallelism. Key frontiers include:

- Extending NAR beyond strict chain- or raster-based orderings to arbitrary graphs and compositional structures, such as arbitrary attention masks and proximity constraints relevant to hierarchical or temporal data.
- Further integrating NAR and AR paradigms, as in Mercury-style think–answer division, to exploit global parallel reasoning with local sequential fidelity [2509.20744].
- Designing more expressive NAR variants capable of capturing target-side dependencies without iterative or AR components, possibly via richer latent variable formulations, alignment mechanisms, or diffusion processes [2204.09269][2509.20744].
- Formalizing information-theoretic and statistical tradeoffs in dependency, parallelism, and data regularity using frameworks like CoMMA [2004.10454].
- Deploying NAR in domains beyond current focus areas, such as code, large-scale knowledge graphs, and symbolic reasoning, especially where neighborhood or graph structure is intrinsic.

A persistent challenge remains: achieving fully-parallel, high-quality NAR generation in settings with inherently strong output dependencies, without reliance on heavy AR teacher supervision or multi-stage inference [2204.09269]. This motivates continued research in model architecture, loss design, and hybrid inference strategies.

---

**Key references:** [2503.10696], [2509.20744], [2110.09596], [2204.09269], [2004.10454], [2206.08317], [2208.12496], [2007.05521], [2402.07373], [2207.10600], [2110.05249]

Source: https://www.emergentmind.com/topics/neighboring-autoregressive-modeling-nar