Next Primitive Prediction (NPP)

• Next Primitive Prediction is a self-supervised technique that forecasts structural building blocks (primitives) to enable abstract, compositional reasoning across modalities.
• It has been applied in text, vision, and algorithmic domains, yielding measurable improvements such as enhanced BLEU scores in text and lower FID in image synthesis.
• The framework leverages curriculum learning and reusable grammar induction to drive interpretability and efficiency without modifying existing transformer or autoregressive architectures.

Next Primitive Prediction (NPP) refers to a class of self-supervised prediction objectives that train models to generate or select the next unit of structure—be it a linguistic phrase, a compositional primitive, or an image patch—rather than the next atomic token or action. By focusing on higher-level or more abstract building blocks, NPP aims to encourage models to acquire reusable semantic units, compositional grammars, or information-rich representations. NPP has been independently instantiated across multiple modalities and tasks, including text auto-completion, open-ended reasoning over algorithmic primitives, and autoregressive image synthesis, each adapting the objective to its domain-specific notion of a “primitive.”

1. Core Definition and Formalization

In its broadest form, NPP generalizes the standard next-token prediction paradigm by shifting the prediction target from elementary symbols to structural or semantic primitives. The precise definition of NPP depends on context:

1. Compositional reasoning: Primitives are atomic computations, subroutines, or motifs from a library $P$. NPP predicts the next primitive $p_{t+1}\in P$ and composition operator $c_{t+1}\in C$ to apply, modeling the construction of solution graphs in a compositional closure $\mathcal{L}(P, C)$. The core loss is

$$\mathcal{L}_{\mathrm{NPP}}(\theta) = -\sum_{t=1}^{T-1} \log p_\theta(p_{t+1}, c_{t+1} | p_{1:t}, c_{1:t}, W),$$

where $W$ encodes world context (Momennejad et al., 13 Jun 2026).

2. Text modeling: Primitives are syntactic phrases (NP, VP, PP). For a prefix $p$ and candidates $S = \{\varphi_1, ..., \varphi_m\}$ extracted from the same sentence, the NPP objective is

$$L_{\mathrm{NPP}}(\theta) = -\sum_i \log p_\theta(\varphi_{i,y_i} | p_i, S_i),$$

where $\varphi_{i,y_i}$ is the correct next phrase (Lee et al., 2021).

3. Vision: Primitives are contiguous patches of discrete image tokens. An image token sequence is partitioned into non-overlapping patches $p_{t+1}\in P$0, each grouping $p_{t+1}\in P$1 tokens. The NPP loss over patch logits $p_{t+1}\in P$2 and ground-truth patch token-ids $p_{t+1}\in P$3 is

$p_{t+1}\in P$4

(Pang et al., 2024).

The unifying property is the model’s explicit learning of transitions between compositional units, fostering the discovery and reuse of regularities above the atomic level.

2. Algorithms and Training Procedures

The practical realization of NPP objectives varies by domain, but several shared schemes recur:

• Primitive Extraction: In text, constituency parsing (e.g., AllenNLP) extracts phrase-level candidates. In open-ended intelligence, primitives are algorithmic routines, while in visual domains, patches are constructed by grouping tokens (e.g., VQGAN embeddings) (Lee et al., 2021, Pang et al., 2024).
• Input Representation: Compositional NPP inputs encode partial solution traces. Text NPP serializes the context and candidate phrases into a sequence. Visual NPP computes mean embeddings per patch and forms a shorter autoregressive sequence (Momennejad et al., 13 Jun 2026, Pang et al., 2024).
• Predictor Update: All implementations train via negative log-likelihood over the next primitive or patch. In compositional settings, optional parsimony regularization (e.g., complexity penalties on primitives) discourages non-reusable or needlessly complex primitives (Momennejad et al., 13 Jun 2026).
• Curriculum and Trace Extraction: In open-ended intelligence, curriculum learning and self-play expand the set of “worlds” (tasks/environments), with each new PTG trace mined for novel or compositional motifs (Momennejad et al., 13 Jun 2026).
• No Architecture Changes: NPP objectives plug into existing transformer or autoregressive architectures without parameter or tokenizer modification (Lee et al., 2021, Pang et al., 2024).

A summary of implementation elements is given below:

Domain	Primitive Type	Extraction/Grouping	Predictor
Text	NP/VP/PP spans	Constituency parser (AllenNLP)	T5-base, sequence-to-sequence
Compositional	Algorithmic routines	Trace mining/subgraph discovery	Transformer w/ PTG encoder
Vision	Image token patches	Grouping + mean embeddings	AR transformer

3. Compositional and Curriculum-Driven Learning

NPP is tightly linked to compositional reasoning and curriculum expansion:

• Compositional Closure: In open-ended intelligence, primitives $p_{t+1}\in P$5 and operators $p_{t+1}\in P$6 generate the potentially infinite closure $p_{t+1}\in P$7. The prediction of $p_{t+1}\in P$8 enables the agent to traverse and expand this space (Momennejad et al., 13 Jun 2026).
• Grammar Acquisition: By focusing on next-step prediction over a library of subroutines, NPP drives emergence of a compact, reusable grammar of solution trajectories and subgraphs.
• Motif Discovery and Library Expansion: Recurring subgraph patterns are wrapped into new, higher-order primitives and appended to $p_{t+1}\in P$9 or $c_{t+1}\in C$0 if they repeatedly appear in high-reward traces (Momennejad et al., 13 Jun 2026).
• Self-Play and World Generation: Agents use their basis $c_{t+1}\in C$1 to mutate “worlds” (environment or problem parameters), forcing continual adaptation and new primitive discovery. Only motifs that generalize across multiple worlds are retained, realizing “transfer as recomposition” (Momennejad et al., 13 Jun 2026).

This curriculum-based expansion and compression of primitive libraries distinguishes NPP from purely token-level next-element prediction.

4. Domain-Specific Instantiations

Compositional Frameworks

In the context of open-ended intelligence, NPP was proposed to enable agents to discover, predict, and recombine algorithmic primitives, supporting transfer to novel and deeper tasks. Empirical studies include:

• Symbolic Derivation in Physics: Agents achieved up to 0.92 compositional depth generalization at test depths 50% greater than in training, compared to 0.23 for a next-token LSTM, by discovering motifs such as “LeastAction→Euler–Lagrange” (Momennejad et al., 13 Jun 2026).
• Evolutionary Biology: NPP yielded a compact primitive set with high reuse (PRI > 0.8), in contrast to traditional program search which produced brittle, task-specific fragments.
• Neuroscience: A neural NPP-like predictor reproduced the emergence of parallel coding in latent context tasks, in agreement with observed hippocampal activity (Momennejad et al., 13 Jun 2026).

Text Generation

For text auto-completion, NPP introduces an intermediate pretraining step focused on phrase-level prediction. The method:

• Uses off-the-shelf constituency parsers to mine NP/VP/PP spans.
• Trains a LLM in a QA-style setup to generate the correct next phrase from a candidate set, fostering richer continuation (Lee et al., 2021).
• Delivers empirical improvements: e.g., in the Enron email domain, NPP-pretrained T5-Base achieves BLEU-4 = 3.2, MetEOR = 7.1, CIDEr = 43.0, SPICE = 4.5, outperforming both next-sentence prediction and next-token models with over triple the parameter count (Lee et al., 2021).

Vision: Autoregressive Image Generation

NPP extends next-token prediction in visuals by aggregating image tokens into patches:

• Each patch groups $c_{t+1}\in C$2 tokens and is embedded by mean pooling.
• Training proceeds in a coarse-to-fine multi-scale regime, starting with large patches (e.g., $c_{t+1}\in C$3) and annealing toward vanilla NTP ($c_{t+1}\in C$4).
• Computational cost is reduced by a factor of ~0.6 while improving output quality: e.g., LlamaGen-XXL with NPP scored FID 2.55 vs 3.10 baseline at 0.53× cost (Pang et al., 2024).
• NPP is domain-agnostic and parameter-free: no new trainable parameters, no custom tokenizers, and full backward compatibility with existing AR model architectures (Pang et al., 2024).

5. Comparative Analysis: NPP vs. Next-Token and Next-Action Prediction

Standard next-token prediction models surface-level dependencies over discrete sequences, while next-action predictors map states to actions typically without explicit grammar induction.

NPP provides several advantages:

• Parsimony Constraint: Enforces the re-use and minimality of primitives, avoiding proliferation of brittle, overfitted fragments (Momennejad et al., 13 Jun 2026).
• Compositional Generalization: The compositional closure supports reasoning over tasks deeper or more structurally novel than those seen in training (Momennejad et al., 13 Jun 2026).
• Role Flexibility: Learned primitives can be rebound or substituted to accommodate changes in world dynamics (e.g., swapping “swim_to” for “move_to” in altered environments).
• Interpretability: PTG traces, phrase choices, or patch sequences reflect meaningful program structures or semantic units that are directly inspectable, facilitating explanation and probe analysis (Momennejad et al., 13 Jun 2026, Lee et al., 2021).
• Empirical Robustness: Across physics, biology, neuroscience, text, and vision, NPP leads to improvements in zero-shot transfer, handling of out-of-distribution generalization, and surface/semantic similarity metrics (Momennejad et al., 13 Jun 2026, Lee et al., 2021, Pang et al., 2024).

6. Evaluation Metrics and Experimental Outcomes

Empirical studies of NPP evaluate generation quality, compositional depth generalization, library reuse, and computational efficiency:

• Text: BLEU-4, METEOR, CIDEr, SPICE—all showing NPP’s consistent +0.2–0.4 gains over strong baselines across email and academic domains (Lee et al., 2021).
• Vision: FID, Inception Score, and Precision/Recall—NPP integration yields FID improvements up to 1.0 and ~40% training FLOPs reduction (Pang et al., 2024).
• Compositional tasks: Compositional Depth Generalization (CDG), Primitive Reuse Index (PRI), and representational parallelism—all favor NPP versus token/action-level methods (Momennejad et al., 13 Jun 2026).

Representative results are summarized:

Domain	Baseline Metric	NPP Metric	Improvement/Impact
Text (Enron)	BLEU-4 = 2.8	3.2	+0.4 vs T5-Base, +2.1 vs GPT-2
Vision (ImageNet)	FID = 3.10	2.55	–0.55, 0.53× compute
Physics (CDG@12)	0.23 (LSTM)	0.92 (NPP)	Order-of-magnitude generalization
Biology (PRI)	0.15 (AST)	>0.8 (NPP)	High primitive reuse

Across all settings, NPP demonstrates structural and statistical superiority in fostering transferable, semantically coherent, and resource-efficient models.

7. Discussion and Future Directions

NPP reframes prediction and reasoning as the generation of the next compositional building block, rather than atomic actions or tokens. Its primary strength is in driving models toward interpretability, modularity, and capacity for open-ended, combinatorial generalization. A plausible implication is the potential for NPP to serve as a unifying training principle for multi-domain agents, facilitating lifelong learning and symbolic abstraction across language, vision, and abstract reasoning. Challenges remain in principled primitive discovery, curriculum orchestration, and adaptive library refinement, but empirical results establish NPP’s role as a robust and extensible objective for structurally aware prediction (Momennejad et al., 13 Jun 2026, Lee et al., 2021, Pang et al., 2024).