---
title: Next Primitive Prediction (NPP)
url: https://www.emergentmind.com/topics/next-primitive-prediction-npp
type: topic
---

# Next Primitive Prediction (NPP)

Next Primitive Prediction (NPP) refers to a class of self-supervised prediction objectives that train models to generate or select the next unit of structure—be it a linguistic phrase, a compositional primitive, or an image patch—rather than the next atomic token or action. By focusing on higher-level or more abstract building blocks, NPP aims to encourage models to acquire reusable semantic units, compositional grammars, or information-rich representations. NPP has been independently instantiated across multiple modalities and tasks, including text auto-completion, open-ended reasoning over algorithmic primitives, and autoregressive image synthesis, each adapting the objective to its domain-specific notion of a “primitive.”

## 1. Core Definition and Formalization

In its broadest form, NPP generalizes the standard next-token prediction paradigm by shifting the prediction target from elementary symbols to structural or semantic primitives. The precise definition of NPP depends on context:

1. **Compositional reasoning**: Primitives are atomic computations, subroutines, or motifs from a library $P$. NPP predicts the next primitive $p_{t+1}\in P$ and composition operator $c_{t+1}\in C$ to apply, modeling the construction of solution graphs in a compositional closure $\mathcal{L}(P, C)$. The core loss is
   $$
   \mathcal{L}_{\mathrm{NPP}}(\theta) = -\sum_{t=1}^{T-1} \log p_\theta(p_{t+1}, c_{t+1} | p_{1:t}, c_{1:t}, W),
   $$
   where $W$ encodes world context [2606.15386].

2. **Text modeling**: Primitives are syntactic phrases (NP, VP, PP). For a prefix $p$ and candidates $S = \{\varphi_1, ..., \varphi_m\}$ extracted from the same sentence, the NPP objective is
   $$
   L_{\mathrm{NPP}}(\theta) = -\sum_i \log p_\theta(\varphi_{i,y_i} | p_i, S_i),
   $$
   where $\varphi_{i,y_i}$ is the correct next phrase [2109.07067].

3. **Vision**: Primitives are contiguous patches of discrete image tokens. An image token sequence is partitioned into non-overlapping patches $p_1, ..., p_P$, each grouping $K$ tokens. The NPP loss over patch logits $\mathrm{Pred}_i$ and ground-truth patch token-ids $\{\mathrm{Index}_k^i\}$ is
   $$
   L_{\mathrm{NPP}} = \frac{1}{N}\sum_{i=1}^P \sum_{k=1}^K \mathrm{CE}(\mathrm{Pred}_i, \mathrm{Index}_k^i)
   $$
   [2412.15321].

The unifying property is the model’s explicit learning of transitions between compositional units, fostering the discovery and reuse of regularities above the atomic level.

## 2. Algorithms and Training Procedures

The practical realization of NPP objectives varies by domain, but several shared schemes recur:

- **Primitive Extraction**: In text, constituency parsing (e.g., AllenNLP) extracts phrase-level candidates. In open-ended intelligence, primitives are algorithmic routines, while in visual domains, patches are constructed by grouping tokens (e.g., VQGAN embeddings) [2109.07067, 2412.15321].
- **Input Representation**: Compositional NPP inputs encode partial solution traces. Text NPP serializes the context and candidate phrases into a sequence. Visual NPP computes mean embeddings per patch and forms a shorter autoregressive sequence [2606.15386, 2412.15321].
- **Predictor Update**: All implementations train via negative log-likelihood over the next primitive or patch. In compositional settings, optional parsimony regularization (e.g., complexity penalties on primitives) discourages non-reusable or needlessly complex primitives [2606.15386].
- **Curriculum and Trace Extraction**: In open-ended intelligence, curriculum learning and self-play expand the set of “worlds” (tasks/environments), with each new PTG trace mined for novel or compositional motifs [2606.15386].
- **No Architecture Changes**: NPP objectives plug into existing transformer or autoregressive architectures without parameter or tokenizer modification [2109.07067, 2412.15321].

A summary of implementation elements is given below:

| Domain      | Primitive Type          | Extraction/Grouping              | Predictor                  |
|-------------|------------------------|----------------------------------|----------------------------|
| Text        | NP/VP/PP spans         | Constituency parser (AllenNLP)   | T5-base, sequence-to-sequence |
| Compositional | Algorithmic routines  | Trace mining/subgraph discovery  | Transformer w/ PTG encoder |
| Vision      | Image token patches    | Grouping + mean embeddings       | AR transformer             |

## 3. Compositional and Curriculum-Driven Learning

NPP is tightly linked to compositional reasoning and curriculum expansion:

- **Compositional Closure**: In open-ended intelligence, primitives $P$ and operators $C$ generate the potentially infinite closure $\mathcal{L}(P,C)$. The prediction of $(p_{t+1},c_{t+1})$ enables the agent to traverse and expand this space [2606.15386].
- **Grammar Acquisition**: By focusing on next-step prediction over a library of subroutines, NPP drives emergence of a compact, reusable grammar of solution trajectories and subgraphs.
- **Motif Discovery and Library Expansion**: Recurring subgraph patterns are wrapped into new, higher-order primitives and appended to $P$ or $C$ if they repeatedly appear in high-reward traces [2606.15386].
- **Self-Play and World Generation**: Agents use their basis $(P,C)$ to mutate “worlds” (environment or problem parameters), forcing continual adaptation and new primitive discovery. Only motifs that generalize across multiple worlds are retained, realizing “transfer as recomposition” [2606.15386].

This curriculum-based expansion and compression of primitive libraries distinguishes NPP from purely token-level next-element prediction.

## 4. Domain-Specific Instantiations

### Compositional Frameworks

In the context of open-ended intelligence, NPP was proposed to enable agents to discover, predict, and recombine algorithmic primitives, supporting transfer to novel and deeper tasks. Empirical studies include:

- **Symbolic Derivation in Physics**: Agents achieved up to 0.92 compositional depth generalization at test depths 50% greater than in training, compared to 0.23 for a next-token LSTM, by discovering motifs such as “LeastAction→Euler–Lagrange” [2606.15386].
- **Evolutionary Biology**: NPP yielded a compact primitive set with high reuse (PRI > 0.8), in contrast to traditional program search which produced brittle, task-specific fragments.
- **Neuroscience**: A neural NPP-like predictor reproduced the emergence of parallel coding in latent context tasks, in agreement with observed hippocampal activity [2606.15386].

### Text Generation

For text auto-completion, NPP introduces an intermediate pretraining step focused on phrase-level prediction. The method:

- Uses off-the-shelf constituency parsers to mine NP/VP/PP spans.
- Trains a language model in a QA-style setup to generate the correct next phrase from a candidate set, fostering richer continuation [2109.07067].
- Delivers empirical improvements: e.g., in the Enron email domain, NPP-pretrained T5-Base achieves BLEU-4 = 3.2, MetEOR = 7.1, CIDEr = 43.0, SPICE = 4.5, outperforming both next-sentence prediction and next-token models with over triple the parameter count [2109.07067].

### Vision: Autoregressive Image Generation

NPP extends next-token prediction in visuals by aggregating image tokens into patches:

- Each patch groups $K = PS^2$ tokens and is embedded by mean pooling.
- Training proceeds in a coarse-to-fine multi-scale regime, starting with large patches (e.g., $P=2,4$) and annealing toward vanilla NTP ($PS=1$).
- Computational cost is reduced by a factor of ~0.6 while improving output quality: e.g., LlamaGen-XXL with NPP scored FID 2.55 vs 3.10 baseline at 0.53× cost [2412.15321].
- NPP is domain-agnostic and parameter-free: no new trainable parameters, no custom tokenizers, and full backward compatibility with existing AR model architectures [2412.15321].

## 5. Comparative Analysis: NPP vs. Next-Token and Next-Action Prediction

Standard next-token prediction models surface-level dependencies over discrete sequences, while next-action predictors map states to actions typically without explicit grammar induction.

NPP provides several advantages:

- **Parsimony Constraint**: Enforces the re-use and minimality of primitives, avoiding proliferation of brittle, overfitted fragments [2606.15386].
- **Compositional Generalization**: The compositional closure supports reasoning over tasks deeper or more structurally novel than those seen in training [2606.15386].
- **Role Flexibility**: Learned primitives can be rebound or substituted to accommodate changes in world dynamics (e.g., swapping “swim_to” for “move_to” in altered environments).
- **Interpretability**: PTG traces, phrase choices, or patch sequences reflect meaningful program structures or semantic units that are directly inspectable, facilitating explanation and probe analysis [2606.15386, 2109.07067].
- **Empirical Robustness**: Across physics, biology, neuroscience, text, and vision, NPP leads to improvements in zero-shot transfer, handling of out-of-distribution generalization, and surface/semantic similarity metrics [2606.15386, 2109.07067, 2412.15321].

## 6. Evaluation Metrics and Experimental Outcomes

Empirical studies of NPP evaluate generation quality, compositional depth generalization, library reuse, and computational efficiency:

- **Text**: BLEU-4, METEOR, CIDEr, SPICE—all showing NPP’s consistent +0.2–0.4 gains over strong baselines across email and academic domains [2109.07067].
- **Vision**: FID, Inception Score, and Precision/Recall—NPP integration yields FID improvements up to 1.0 and ~40% training FLOPs reduction [2412.15321].
- **Compositional tasks**: Compositional Depth Generalization (CDG), Primitive Reuse Index (PRI), and representational parallelism—all favor NPP versus token/action-level methods [2606.15386].

Representative results are summarized:

| Domain              | Baseline Metric  | NPP Metric     | Improvement/Impact                |
|---------------------|-----------------|---------------|-----------------------------------|
| Text (Enron)        | BLEU-4 = 2.8    | 3.2           | +0.4 vs T5-Base, +2.1 vs GPT-2    |
| Vision (ImageNet)   | FID = 3.10      | 2.55          | –0.55, 0.53× compute              |
| Physics (CDG@12)    | 0.23 (LSTM)     | 0.92 (NPP)    | Order-of-magnitude generalization  |
| Biology (PRI)       | 0.15 (AST)      | >0.8 (NPP)    | High primitive reuse              |

Across all settings, NPP demonstrates structural and statistical superiority in fostering transferable, semantically coherent, and resource-efficient models.

## 7. Discussion and Future Directions

NPP reframes prediction and reasoning as the generation of the next compositional building block, rather than atomic actions or tokens. Its primary strength is in driving models toward interpretability, modularity, and capacity for open-ended, combinatorial generalization. A plausible implication is the potential for NPP to serve as a unifying training principle for multi-domain agents, facilitating lifelong learning and symbolic abstraction across language, vision, and abstract reasoning. Challenges remain in principled primitive discovery, curriculum orchestration, and adaptive library refinement, but empirical results establish NPP’s role as a robust and extensible objective for structurally aware prediction [2606.15386, 2109.07067, 2412.15321].

Source: https://www.emergentmind.com/topics/next-primitive-prediction-npp