Next Editing-token Prediction

• Next Editing-token Prediction (NEP) is an autoregressive framework that regenerates only tokens in user-specified image regions while preserving the original context.
• NEP leverages VQGAN tokenization, binary mask embeddings, and any-order autoregression via RLlamaGen to achieve efficient, targeted image editing.
• This approach shifts from full-image reconstruction to localized predictive learning, enabling zero-shot editing and iterative refinement for various visual tasks.

Searching arXiv for the primary NEP paper and closely related predictive-learning work to ground the article. arXiv search: (Wu et al., 8 Aug 2025) "NEP: Autoregressive Image Editing via Next Editing Token Prediction" — found. arXiv search: (Xu et al., 18 Dec 2025) "Next-Embedding Prediction Makes Strong Vision Learners" — found. arXiv search: (Yao et al., 17 May 2026) "Rethinking Point Clouds as Sequences: A Causal Next-Token Predictive Learning Framework" — found. Next Editing-token Prediction (NEP) is an autoregressive formulation of text-guided image editing in which only the tokens inside designated editing regions are regenerated, while non-editing tokens are copied from the source image unchanged. In the formulation introduced by “NEP: Autoregressive Image Editing via Next Editing Token Prediction,” an image is tokenized by a VQGAN, editing regions are specified by a binary mask, and the model predicts the next token only over the subset of positions marked for change, conditioned on the text instruction, the full source image tokens, and mask embeddings (Wu et al., 8 Aug 2025). This reframes image editing from full-image resynthesis to conditional, localized regeneration, and places it within a broader family of predictive-learning methods that replace reconstruction-heavy objectives with autoregressive or latent predictive ones.

1. Formal definition and token-level semantics

In a standard autoregressive text-to-image model, an image token sequence $I=\{I_1,\dots,I_L\}$ is generated in a fixed order by next-token prediction,

$$p(I \mid T)=\prod_{i=1}^{L} p(I_i \mid I_{1,\dots,i-1}; T).$$

NEP retains the autoregressive factorization but restricts prediction to a masked subset of positions. An “editing token” is any image token whose location falls inside the editing region after the pixel-level binary mask $M \in \{0,1\}^{H \times W}$ is downsampled to the token grid via non-overlapping max-pooling and flattened as $M^E=\{m_1,\dots,m_L\}$, with $m_i\in\{0,1\}$. Tokens with $m_i=1$ are regenerated; tokens with $m_i=0$ remain fixed (Wu et al., 8 Aug 2025).

A central point is that editing tokens are not a new discrete code type. They remain ordinary VQGAN codebook indices, but the model is informed which positions are editable through a parallel mask-embedding sequence. This distinguishes NEP from masked-token models that operate in parallel and from standard autoregressive image models that regenerate the entire image. Formally, if $I^S_{1,\dots,L}$ denotes source image tokens, $I_E=\{I_{o^E_1},\dots,I_{o^E_{L_E}}\}$ the editable subset, and $O^E=\{o^E_1,\dots,o^E_{L_E}\}$ their positions, then NEP models the conditional distribution over only those positions: $$p(I \mid T)=\prod_{i=1}^{L} p(I_i \mid I_{1,\dots,i-1}; T).$$0 The resulting edited token map is assembled by replacing editable positions with predicted tokens and copying all others from the source image.

Two misconceptions are explicitly excluded by the formulation. First, NEP is not full-image next-token prediction with a mask appended; the number of predicted outputs is $$p(I \mid T)=\prod_{i=1}^{L} p(I_i \mid I_{1,\dots,i-1}; T).$$1, not $$p(I \mid T)=\prod_{i=1}^{L} p(I_i \mid I_{1,\dots,i-1}; T).$$2. Second, NEP is not a generic masked-image-model objective; it is an autoregressive process over an arbitrary subset of positions, and the generation order can be raster or user-specified (Wu et al., 8 Aug 2025).

2. Any-order autoregression and the RLlamaGen foundation

NEP depends on an any-order autoregressive text-to-image backbone, because a raster-only decoder cannot naturally regenerate an arbitrary subset of positions. The paper introduces RLlamaGen, derived from LlamaGen, as that foundation. RLlamaGen uses a VQGAN image tokenizer, a FLAN-T5 text encoder whose projected embeddings are left-padded to fixed length $$p(I \mid T)=\prod_{i=1}^{L} p(I_i \mid I_{1,\dots,i-1}; T).$$3, and a Llama-like transformer decoder backbone instantiated as LlamaGen-XL with 775M parameters (Wu et al., 8 Aug 2025).

The key modification is random-order generation with order-aware positional embeddings. During training, the model samples a random permutation $$p(I \mid T)=\prod_{i=1}^{L} p(I_i \mid I_{1,\dots,i-1}; T).$$4 of token indices and predicts image tokens in that order, while adding learned positional embeddings $$p(I \mid T)=\prod_{i=1}^{L} p(I_i \mid I_{1,\dots,i-1}; T).$$5 that encode token position under arbitrary generation order. The objective remains token-level cross-entropy, but every sequence prefix now corresponds to an arbitrary partial image rather than a raster prefix. This teaches the model to condition on arbitrary subsets of tokens and to fill in the remainder in arbitrary orders, which is the capability NEP requires for region-selective editing (Wu et al., 8 Aug 2025).

This any-order design is also what permits zero-shot editing before any editing-specific fine-tuning. The paper reports that RLlamaGen, trained only on text-to-image data with random-order generation, can already regenerate masked regions conditioned on the remaining tokens and the text instruction. Qualitative examples include adding an object, changing color, and changing object state. NEP fine-tuning then specializes this latent capability into an explicit region-aware editing model (Wu et al., 8 Aug 2025).

From a broader methodological perspective, RLlamaGen aligns NEP with a larger predictive-learning trend in which causal dependency modeling over tokens or embeddings is treated as a modality-agnostic primitive. In vision, NEPA predicts future patch embeddings with causal masking and stop-gradient rather than reconstructing pixels or discrete visual tokens (Xu et al., 18 Dec 2025). In point clouds, PointNTP serializes local patches and applies a fully causal, decoder-free latent next-token objective (Yao et al., 17 May 2026). This suggests that NEP’s any-order subset autoregression is part of a wider shift from reconstruction-centered objectives to predictive dependency modeling.

3. Mask-aware conditioning, inference, and adaptation

The editing model consumes three input streams. The first is the text instruction $$p(I \mid T)=\prod_{i=1}^{L} p(I_i \mid I_{1,\dots,i-1}; T).$$6, encoded as FLAN-T5 embeddings projected into the autoregressive hidden dimension. The second is the source image token sequence $$p(I \mid T)=\prod_{i=1}^{L} p(I_i \mid I_{1,\dots,i-1}; T).$$7, obtained from the VQGAN encoder and used only as conditioning. The third is a mask-embedding sequence $$p(I \mid T)=\prod_{i=1}^{L} p(I_i \mid I_{1,\dots,i-1}; T).$$8, derived from the downsampled binary mask $$p(I \mid T)=\prod_{i=1}^{L} p(I_i \mid I_{1,\dots,i-1}; T).$$9. Each position receives one of two learned embeddings: $M \in \{0,1\}^{H \times W}$0 This two-code mask codebook has only 3.6k trainable parameters, and the full input length becomes $M \in \{0,1\}^{H \times W}$1: text tokens plus source-image tokens and mask embeddings (Wu et al., 8 Aug 2025).

At inference time, NEP proceeds by encoding the instruction, source image, and mask, identifying the editable token indices $M \in \{0,1\}^{H \times W}$2, and then autoregressively predicting only those positions. Each prediction conditions on the full source token grid, the full mask sequence, the text instruction, and previously generated editing tokens with their positional embeddings. The loop terminates after $M \in \{0,1\}^{H \times W}$3 steps, after which the predicted tokens are merged back into the original token map and decoded by the VQGAN decoder. If no mask is provided and the edit is global, the method degenerates to ordinary next-token prediction over all positions (Wu et al., 8 Aug 2025).

The region specification is pixel-level and arbitrary in shape. Because the binary mask is downsampled by max-pooling to token resolution, any patch containing at least one edited pixel becomes editable. This allows multiple disjoint regions and both small and large areas. The claimed computational advantage follows directly: the cost scales with $M \in \{0,1\}^{H \times W}$4 rather than $M \in \{0,1\}^{H \times W}$5, since only editable tokens are decoded (Wu et al., 8 Aug 2025).

The fine-tuning stage uses UltraEdit, which contains 4M image pairs, 131k of them with region annotations. Samples without masks are treated as full-image generation cases. Fine-tuning adds only the two mask embeddings, preserving the autoregressive backbone. The paper reports 3.9M fine-tuning steps with batch size 100 and learning rate $M \in \{0,1\}^{H \times W}$6 on 4×A100 GPUs, plus 2k additional steps at $M \in \{0,1\}^{H \times W}$7 resolution; for Emu Edit, it uses learning rate $M \in \{0,1\}^{H \times W}$8 for 60k steps (Wu et al., 8 Aug 2025).

A useful interpretive distinction is drawn around “zero-shot.” In one sense, it refers to RLlamaGen’s editing ability before any editing-specific fine-tuning. In another, it refers to NEP’s ability to handle instructions and regions not explicitly annotated during training, such as free-form instructions in Emu Edit where no explicit masks are provided (Wu et al., 8 Aug 2025).

4. Benchmarks, efficiency, and test-time scaling

The paper evaluates NEP on three benchmark families.

Benchmark	Setting	Metrics
MagicBrush	Region-aware editing, single-turn and multi-turn	L1, L2, CLIP-I, DINO
Emu Edit	Free-form editing, no ground-truth targets	CLIPout, CLIPdir, L1, CLIPimg, DINO
COCO-30K	Text-to-image	FID, CLIP score

On MagicBrush, NEP is reported as achieving the best L1, L2, CLIP-I, and DINO in single-turn evaluation, with single-turn L1 $M \in \{0,1\}^{H \times W}$9 and CLIP-I $M^E=\{m_1,\dots,m_L\}$0. On multi-turn MagicBrush, it is competitive or best on most metrics. On Emu Edit, it achieves the best or near-best CLIPout, CLIPdir, CLIPimg, and DINO among state-of-the-art baselines, even without explicit masks (Wu et al., 8 Aug 2025).

Ablation against a full-image next-token baseline directly supports the subset-autoregression claim. On multi-turn MagicBrush, NEP, which predicts only $M^E=\{m_1,\dots,m_L\}$1 tokens, reaches L1 $M^E=\{m_1,\dots,m_L\}$2 and CLIP-I $M^E=\{m_1,\dots,m_L\}$3, whereas the NTP variant that predicts all $M^E=\{m_1,\dots,m_L\}$4 tokens reaches L1 $M^E=\{m_1,\dots,m_L\}$5 and CLIP-I $M^E=\{m_1,\dots,m_L\}$6. The paper interprets this as evidence that explicit region selection reduces the learning bias toward reconstructing non-editing regions (Wu et al., 8 Aug 2025).

Efficiency is presented as a second major advantage. The paper states that full-image editing methods incur unnecessary computational cost because they resynthesize all pixels or tokens even when only a local region changes. NEP instead copies non-editing tokens directly. In Table 5, inference time on MagicBrush is reported as 2.88s for NEP, compared with 2.94s for UltraEdit and 10.70s for EditAR, while also noting that NEP uses more GPU memory due to the longer conditioning sequence (Wu et al., 8 Aug 2025).

NEP also supports test-time scaling through iterative refinement. The procedure is: generate an initial image, score it with an image reward model such as ImageReward, identify the $M^E=\{m_1,\dots,m_L\}$7 lowest-scoring or most improvable tokens, define them as a revision region, regenerate them with NEP in multiple random orders, accept revisions that improve the reward, and repeat. On COCO-30K, starting from RLlamaGen without post-training, the paper reports CLIP score $M^E=\{m_1,\dots,m_L\}$8 and FID $M^E=\{m_1,\dots,m_L\}$9; after one revision round, CLIP $m_i\in\{0,1\}$0 and FID $m_i\in\{0,1\}$1; after 2–4 rounds, CLIP saturates at $m_i\in\{0,1\}$2 while FID improves further to $m_i\in\{0,1\}$3. It also reports, relative to original LlamaGen, a CLIP improvement from $m_i\in\{0,1\}$4 to $m_i\in\{0,1\}$5 and FID improvement from $m_i\in\{0,1\}$6 to $m_i\in\{0,1\}$7 under NEP-based test-time scaling (Wu et al., 8 Aug 2025).

5. Position within predictive-learning research and adjacent usages

NEP sits within a broader class of methods that reinterpret generation, representation learning, or structured prediction as causal next-token or next-embedding prediction. In images, NEPA trains a ViT with causal masking and stop-gradient to predict the next patch embedding, with no pixel reconstruction, discrete tokenizer, or task-specific pretraining head, reaching 83.8% and 85.3% top-1 ImageNet-1K accuracy for ViT-B and ViT-L after fine-tuning (Xu et al., 18 Dec 2025). In point clouds, PointNTP reformulates self-supervised learning as fully causal, decoder-free latent next-token prediction on serialized patch embeddings and reports 93.8%, 92.6%, and 89.3% on the three ScanObjectNN variants listed in the paper (Yao et al., 17 May 2026). In language modeling, ContextLM augments standard token prediction with a latent next-context prediction pathway over fixed-size token chunks while keeping ordinary autoregressive evaluation intact (Dai et al., 23 Oct 2025).

These works are not image editing methods, but they clarify NEP’s conceptual placement. NEP predicts a subset of image tokens rather than future patch embeddings, future context embeddings, or future point-cloud patches. A plausible implication is that subset autoregression in image editing and latent predictive learning in representation-focused settings are parallel responses to the same limitation: objectives centered on reconstruction or full-sequence regeneration can underemphasize dependency structure and local causal relevance.

The acronym “NEP” is not unique across fields. In video reasoning, it denotes Next-Event Prediction, where a multimodal model predicts textual summaries of future video events from past frames (Wang et al., 28 May 2025). In software engineering, “Next Edit Prediction” or “Next Edit Suggestion” refers to proactive prediction of edit location and content from code state and interaction history rather than region-selective image regeneration (Lu et al., 13 Aug 2025, Chen et al., 4 Aug 2025). Those usages are task-distinct, but they share the underlying intuition that predicting the next structured change can be more natural than reconstructing an entire output.

6. Limitations, failure modes, and outlook

The paper identifies dependence on user-provided masks as the main explicit limitation. Precise region editing currently requires masks, which implies extra annotation effort or reliance on external segmentation models such as SAM or LISA, as well as additional computation. Future work is described in terms of automated or unified region localization, including inferring editing regions from text or learned attention (Wu et al., 8 Aug 2025).

Other limitations are presented more implicitly. Large structural changes may erode the benefits of local-only regeneration, because they require more global synthesis. Domain shift is also a concern: the model is trained primarily on web-scale text-to-image data and UltraEdit, so extreme domains such as medical or scientific imagery may degrade performance. Memory usage remains nontrivial because the model conditions on source tokens and mask embeddings in addition to text, even though the number of predicted steps is reduced (Wu et al., 8 Aug 2025).

A further misconception addressed by the paper is that any-region editing implies annotation-free localized editing. That is not claimed. Rather, the current system supports arbitrary regions once a mask exists, while free-form editing without explicit masks is handled by falling back to global generation or by relying on the model’s zero-shot generalization (Wu et al., 8 Aug 2025).

The broader trajectory suggested by NEP is the treatment of editing as subset autoregression rather than holistic regeneration. The paper itself names automated region selection, stronger test-time verifiers, and broader examination of test-time scaling as immediate directions. It also lists possible extensions to video editing, 3D and multi-view editing, and multimodal editing, though these are explicitly not implemented. Taken together, these directions suggest that NEP is best understood not merely as a benchmark improvement, but as a specific autoregressive principle: predict only the tokens whose causal status is “to be changed,” while preserving the remainder as immutable context (Wu et al., 8 Aug 2025).