---
title: NARRepair Model – Non-Autoregressive APR
url: https://www.emergentmind.com/topics/narrepair-model
type: topic
---

# NARRepair Model – Non-Autoregressive APR

The NARRepair Model is a non-autoregressive deep learning framework custom-built for automatic program repair (APR). Unlike conventional autoregressive approaches that generate patches token-by-token in sequence, NARRepair constructs program repairs in parallel, thereby achieving considerable reductions in inference latency while delivering state-of-the-art repair accuracy. The architecture integrates a repair action predictor, an inter-token dependency extractor based on source code ASTs, and a specialized two-stage decoder to address inherent challenges in parallel code generation. NARRepair has been validated across multiple APR benchmarks, demonstrating substantial speed advantages and high patch fidelity compared to both autoregressive and large language model–based systems [2510.01825][2406.16526].

## 1. Architectural Innovations

NARRepair's design centers around three core modules:

### 1. Repair Action Predictor

Rather than directly modeling the final code token at each position—a high-dimensional, error-prone task—NARRepair predicts a repair action for every token in the buggy code. The repair actions are classified as “keep” (retain the token), “replace” (substitute with a new token), “insert” (add tokens), and “delete” (remove the token). This prediction occurs jointly with the expected repair length for each location (e.g., “keep” and “replace” actions typically receive length 1, “delete” is length 0). The predictor leverages a convolutional network atop encoded token features, followed by fully connected layers. Formally:

\[
\text{Action}_i=\text{Linear}_1(\text{Dropout}(\text{ReLU}(\text{ConV}(E_{i:n}))))
\]
\[
\text{Length}_i=\text{Linear}_2(\text{Dropout}(\text{ReLU}(\text{ConV}(E_{i:n}))))
\]

The categorical losses for training are:
\[
L_{\text{act}} = -\sum_i \log p_{\text{act}}(a_i|X,N)
\]
\[
L_{\text{length}} = -\sum_i \log p_{\text{length}}(l_i|X,M)
\]
where $X$ is the input, $N$ and $M$ are model parameters.

By reducing prediction space to four discrete actions and associated lengths, NARRepair maximally preserves correct segments and sharply limits unnecessary modifications.

### 2. Inter-Token Dependency Extraction

Traditional non-autoregressive decoders lack explicit modeling of sequential dependencies, a critical weakness for syntax-sensitive domains like program repair. NARRepair augments token-level representations with structural dependency information derived from the program's Abstract Syntax Tree (AST). Using a program analysis tool (e.g., Tree-sitter), the nearest common parent node in the AST for every token pair is encoded as a dependency matrix. This structural information is integrated into token features using cross-attention:

\[
Q = W^{q} D_{1:m}, \quad K = W^{k} D_{1:m}
\]
\[
M_{\text{dependency}} = \text{Linear3}(Q) K^\top
\]
\[
\text{score} = \text{softmax}\left(\frac{Q K^\top}{\sqrt{d}}\right)
\]
\[
V = W^{v} D_{1:m}
\]
\[
H_{1:m} = \sum_i \text{score} \cdot V
\]

This mechanism introduces AST-derived dependencies into the representations powering both action prediction and decoding, substantially enhancing syntactic and semantic consistency.

### 3. Two-Stage Decoding

To alleviate deficits in contextual representation intrinsic to parallel decoding, NARRepair employs a masked two-stage decoder:

- **Stage 1:** All tokens are decoded in parallel, producing a patch and associated confidence scores.
- **Stage 2:** Low-confidence tokens (either by prediction probability or repair action) are masked, and the decoder is rerun in a masked language-model fashion to refine uncertain regions using available context.

Let $P_{\text{first}} = \text{softmax}(\text{Decoder}_{\text{first}}(H_{1:m}))$ denote initial probabilities. The mask function $f_{\text{mask}}$ replaces low-confidence entries with a [MASK] token. The second pass computes $P_{\text{second}} = \text{softmax}(\text{Decoder}_{\text{second}}(f_{\text{mask}}(E_{\text{first}})))$. The final loss used in training is

\[
L_{\text{dec}} = -\sum_i \log\left(P_{\text{dec}}(R | H, \partial)\right)
\]
\[
L = L_{\text{dec}} + \alpha(L_{\text{act}} + L_{\text{length}}) + \lambda L_{\text{depend}}
\]
where $\alpha$ and $\lambda$ are module balancing coefficients (typically 0.1).

## 2. Integration of Non-Autoregressive Generation and APR Constraints

In contrast to autoregressive models that generate target patches token-by-token in left-to-right order (with latent representations accumulating sequential context), NARRepair's non-autoregressive approach supports fully parallel generation. This is only feasible through the action prediction strategy, dependency injection from AST structures, and subsequent contextual refinement. Such integration is required because program repair tasks are extremely sensitive to syntactic correctness, semantic preservation, and localized patching. Each module is architected to directly address characteristic deficits of naive fully-parallel generative models.

## 3. Empirical Evaluation

NARRepair has been thoroughly evaluated on three benchmark datasets:

| Dataset           | # Bugs | NARRepair (3min limit) | AR-based SOTA | GPU Speedup |
|-------------------|--------|-----------------------|---------------|-------------|
| Defects4J v1.2    | 395    | Up to 35              | Lower         | 1.4–6.4×    |
| Defects4J v2.0    | 420+   | Higher                | Lower         |             |
| QuixBugs          | 40     | Higher                | Lower         |             |

NARRepair outperforms baselines—including autoregressive and sequence-to-sequence LLM-based systems—in both repair speed and patch success rate under realistic time constraints [2510.01825][2406.16526]. Its accuracy remains robust under imperfect fault localization and short inference intervals, a critical requirement in practical deployment settings.

## 4. Impact on Program Repair Research and Practice

The introduction of NARRepair addresses previously limiting trade-offs in APR research between patch quality and inference latency. The approach enables rapid, high-quality bug fixing suitable for latency-sensitive applications such as interactive development environments, CI/CD pipelines, and embedded system maintenance. The modular innovation—action-based patching, AST-driven dependency representation, and staged decoding—sets a new direction for integrating explicit syntactic knowledge and flexible generative architectures in program repair. Speed improvements of 1.4–6.4× are typical, with evidence of even greater gains on larger models and hardware acceleration [2510.01825].

## 5. Relationship to Related Model Repair Frameworks

NARRepair differs fundamentally from methods such as static analysis-driven repair, graph transformation approaches, or neuron-level LM editing. The focus is on non-autoregressive generative paradigms within the APR domain. While abstraction-driven model repair (e.g., KMTS/CTL repair) [1506.06165] and rule-based graph repair [1912.09610] offer guarantees and systematic synthesis for generalized system models, they do not address the specific computational and accuracy constraints of code patch synthesis via deep learning, nor do they scale to real-time patch generation tasks. NARRepair’s use of ASTs as dependency scaffolding is tailored for syntactic programming languages and is orthogonal to approaches centered on semantic subtyping or symbolic reasoning.

## 6. Future Directions

The core innovations of NARRepair—parallelized patch synthesis via action prediction, AST-guided dependency extraction, and iterative context refinement—suggest several research opportunities. These include extending the framework for multi-hunk and cross-file repairs, integrating upstream fault localization signals, and merging with semantics-based neuron editing or symbolic verification for correctness guarantees. Additionally, the approach may generalize to repair tasks in compiled, low-level, or dynamically typed languages where localized patching is essential.

## 7. Key Formula Table

| Component                | Formula / Expression                                                                                                                                     | Description                                      |
|--------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------|
| Encoder                  | $E_{i:n} = \text{Encoder}(W_{i:n} + W_{\text{pos}})$                                                                                                     | Token encoding                                   |
| Repair Action Prediction | $X_{\text{feature}} = \text{ConV}(E_{1:n})$<br>$\text{Act}_{1:n} = \text{Linear}_1(\text{Dropout}(\text{ReLU}(X_{\text{feature}})))$<br>$\text{Len}_{1:n} = \text{Linear}_2(\text{Dropout}(\text{ReLU}(X_{\text{feature}})))$ | Predicting repair action and length              |
| Dependency Extraction    | $Q = W^q D_{1:m}$<br>$K = W^k D_{1:m}$<br>$M_{\text{dependency}} = \text{Linear3}(Q) K^\top$<br>$H_{1:m} = \sum \text{score} \cdot (W^v D_{1:m})$        | AST-based inter-token dependency                 |
| Two-Stage Decoding       | $E_{\text{first}} = \text{Decoder}_{\text{first}}(H_{1:m})$<br>$E_{\text{mask}} = \text{MaskFunc}(E_{\text{first}})$<br>$E_{\text{second}} = \text{Decoder}_{\text{second}}(E_{\text{mask}})$ | Parallel preliminary + masked contextual pass    |
| Loss                     | $L = L_{\text{dec}} + \alpha (L_{\text{act}} + L_{\text{length}}) + \lambda L_{\text{depend}}$                                                          | Joint module loss with balancing coefficients    |

These architectural components collectively deliver state-of-the-art repair accuracy and repair latency in APR.

## References

- "Towards Speeding up Program Repair with Non-Autoregressive Model" [2510.01825]
- "NARRepair: Non-Autoregressive Code Generation Model for Automatic Program Repair" [2406.16526]

Source: https://www.emergentmind.com/topics/narrepair-model