Next-Concept Modeling (NCM) - Overview

• NCM predicts abstract, semantic, or structural units (such as intent or concept sets) following observed contexts, offering broader predictive scope than next-token prediction.
• Applications of NCM include language modeling, recommendation systems, and software model completion, with examples like AlipayKG for intent prediction and Mimir for sentence embeddings.
• Key NCM models include Semantic and intent concept representation (like AlipayKG) and Learned latent concepts, with practical use in next-basket recommendation and multi-intent generation

Next-Concept Modeling (NCM) is a family of sequence-modeling approaches that predicts an abstract, semantic, latent, structural, or otherwise higher-level unit that follows an observed context, rather than predicting only the next surface token or local event. The predicted unit may be a user intent, concept set, sentence embedding, quantized multi-token representation, software-model location, or multiscale discrete code. NCM therefore does not denote one canonical architecture or objective. Its common principle is to shift prediction toward a representation with broader temporal scope, greater semantic abstraction, or stronger structural relevance, often while retaining a lower-level generation or completion mechanism.

1. Definition, scope, and conceptual foundations

In a generic sequential formulation, NCM estimates a conditional distribution or predictive representation of a future concept:

$P(C_{t+1}\mid C_{1:t},A_{1:t},\text{context}),$

where $C_t$ denotes a concept observed at step $t$, and $A_t$ denotes dynamic contextual information. A concept may be represented as a singleton, a set, an ordered sequence, a sentence-level embedding, a latent vector, a model element, or a hierarchy of discrete codes.

The central distinction from next-token prediction (NTP) is the granularity of the prediction target. NTP models a surface continuation such as a tokenizer symbol. NCM instead predicts a unit that can span multiple tokens, aggregate several events, summarize a semantic state, or identify a future structural focus. The relationship may be hierarchical:

$$\text{context} \rightarrow \text{next concept} \rightarrow \text{surface realization}.$$

The term concept is operationally domain-dependent. In “Beyond Tokens: Concept-Level Training Objectives for LLMs,” concepts are groups of lexical realizations, such as “mom,” “mommy,” and “mother,” treated as approximately equivalent (Iyer et al., 16 Jan 2026). In ConceptLM and NCP-ArchPreview, concepts are learned latent representations obtained by pooling contiguous hidden states and quantizing them with product vector quantization; they are not necessarily human-interpretable semantic categories (Liu et al., 9 Feb 2026, Team et al., 9 Sep 2026). In Mimir, each complete sentence is treated as one concept and represented by a continuous SONAR embedding (Musacchio et al., 24 May 2026). In multiscale NCM, concepts are discrete codes at progressively finer representational levels (Kumbong et al., 9 Oct 2026).

Consequently, NCM should be distinguished from several related ideas. It is not necessarily a semantic classifier, because concepts may be latent and learned rather than externally annotated. It is not necessarily a replacement for token generation, since many systems jointly optimize concept prediction and NTP. It is not necessarily single-step prediction: outputs may be ranked sets, multilevel code sequences, or iterative focus locations. Finally, NCM does not require a discrete concept vocabulary; Mimir uses continuous sentence embeddings, whereas ConceptLM, NCP-ArchPreview, and multiscale NCM use discrete or product-quantized representations.

2. Major forms of concept representation

Semantic and intent concepts

A knowledge-grounded NCM system can normalize heterogeneous events into an explicit concept vocabulary. AlipayKG introduces an ontology containing users, items, scenarios, and intents, with relations such as $\text{need}$ and $\text{provide}$. A movie-ticket bill can be mapped to “buy movie tickets,” while a search query can be mapped to an intent such as “find a Sichuan-style hot-pot restaurant” (He et al., 2023).

The intent layer distinguishes Industry Category, Primitive Intent, Property Value, and Composite Intent. Primitive intents commonly follow a functional-verb-plus-product-entity structure, such as “order coffee” or “buy train tickets.” Composite intents combine a primitive intent with an attribute-value relation, for example:

$$\text{Order coffee delivery} = (\text{order coffee},\text{service type},\text{delivery}).$$

The ontology also contains $\text{isA}$ relations, enabling coarse-to-fine representations such as “buy iPhone 13,” “buy mobile phone,” and a broader shopping category. These structures provide semantic targets and graph priors for a future next-intent predictor, although AlipayKG itself primarily describes knowledge-graph construction, item–intent matching, intent classification, search, recommendation, and risk-management applications rather than a formally reported sequential NCM model (He et al., 2023).

User-defined and personalized concepts

Agile Modeling treats a concept as a user-specific visual category rather than a crowd-defined class. A user supplies positive and negative phrases, retrieves candidate images using CLIP or ALIGN, labels a small set of images, and trains a compact classifier with active learning (Stretcu et al., 2023). The resulting concept is the decision boundary induced by that user’s labels.

This framework provides concept elicitation and iterative refinement rather than explicit next-concept generation. Its relevance to NCM lies in the possibility of modeling a newly introduced, refined, or temporally updated concept. The system selects the next informative examples to label through uncertainty sampling, but it does not itself determine which semantic concept should follow another.

Lexical equivalence classes

Concept-aware language-model training can group multiple surface forms into a target concept. The synonym example is:

$$\{\text{mom},\text{mommy},\text{mother}\} \mapsto \text{MOTHER}.$$

The corresponding methods use WordNet or context-aware Llama-3-8B-Instruct prompting to construct synonym or hypernym alternatives. NCP data augmentation trains on multiple substituted sentences, while an NCP loss rewards probability assigned to a set of acceptable completions (Iyer et al., 16 Jan 2026). These methods remain token-generating: they do not introduce an explicit concept decoder or a separate autoregressive concept sequence.

Continuous sentence concepts

Mimir defines a concept as one complete sentence. A document or conversation is segmented into sentences, each sentence is encoded with SONAR, and a Two-Tower diffusion LCM predicts the embedding of the next sentence. The predicted embedding is decoded back into text and fed into the context for subsequent prediction (Musacchio et al., 24 May 2026).

Mimir therefore uses continuous semantic-space regression rather than a discrete concept vocabulary. Its objective is mean-squared error between predicted and target SONAR embeddings:

$C_t$0

Learned latent concepts

ConceptLM and NCP-ArchPreview construct concepts online from hidden states. Contiguous groups of $C_t$1 token states are mean-pooled:

$C_t$2

The resulting vectors are divided into segments and quantized with product codebooks. A concept is therefore a tuple of codebook selections, with effective vocabulary size $C_t$3. A dedicated Concept Module predicts the next codebook distributions, forms an expected codeword embedding, and feeds the predicted representation back into token generation (Liu et al., 9 Feb 2026, Team et al., 9 Sep 2026).

Multiscale residual concepts

The multiscale NCM framework uses a residual VQ-VAE tokenizer. A sequence is represented by levels $C_t$4, whose code lengths satisfy:

$C_t$5

Early levels are short and coarse, while later levels add residual detail. The transformer predicts all codes within a level in parallel, then uses the cumulative reconstruction to condition the next level (Kumbong et al., 9 Oct 2026). Here, concepts are not necessarily temporal units; they are progressively refined representations of the same complete sequence.

3. Architectural mechanisms

Separate dynamic streams

AnDa provides an architecture for next-basket recommendation that transfers naturally to concept prediction. It separately encodes basket-item sequences, categorical dynamic attributes, and numerical dynamic attributes with causal multi-head self-attention. Dynamic attributes may evolve asynchronously, exhibit recency and periodicity, and reflect changing user interests more directly than static embeddings (Chen et al., 2021).

For NCM, the corresponding streams may include prior concepts, user state, external context, numerical measurements, and event metadata. Separate encoders preserve stream-specific temporal structure before fusion. Time-aware padding retains the actual position of missing observations, while periodic-index embeddings expose recurring temporal patterns. The architecture also applies intra-step attention to items or attributes occurring together, which transfers to concept co-occurrence, prerequisite, compatibility, or multi-intent modeling.

Knowledge-grounded concept interfaces

AlipayKG supplies a semantic interface between heterogeneous raw events and concept sequences. Its construction pipeline includes word segmentation, part-of-speech tagging, rule filtering, TF–IDF ranking, threshold filtering, crowdsourced labeling, and binary classification. Item–intent matching combines text and image representations, while intent embeddings are obtained with Node2Vec over an Intent Graph containing categories, intents, functional verbs, product entities, and HowNet sememes (He et al., 2023).

This approach addresses a problem preceding sequence prediction: defining the target concept space and mapping raw observations into it. The paper does not specify a Transformer that predicts $C_t$6 from $C_t$7, nor does it report next-concept metrics. Its direct contribution to NCM is therefore concept normalization, hierarchical structure, graph representation, and explainable grounding.

Latent concept modules

ConceptLM uses a token-level encoder, a compressed concept-level Transformer, and a token-level decoder. The concept-level module predicts a weighted combination of codebook vectors for each product-quantization segment. The complete objective is:

$C_t$8

The predicted concept is shifted and broadcast over token positions, preventing future leakage, then added to token hidden states.

NCP-ArchPreview scales this design to approximately $C_t$9B parameters. Its architecture contains a 16-layer Token Encoder, an 8-layer Concept Module, and a 16-layer Token Decoder. It additionally uses intra-module and cross-module hierarchical residual routing, including Token Encoder $t$0 Concept Module, Token Encoder $t$1 Token Decoder, and Concept Module $t$2 Token Decoder pathways (Team et al., 9 Sep 2026).

Coarse-to-fine generation

Multiscale NCM separates representation learning from generation. The VQ-VAE is first trained and frozen. The generator then predicts level $t$3, reconstructs its contribution, predicts level $t$4, and continues through level $t$5. All codes within a level are sampled in parallel:

$t$6

This yields $t$7 sequential transformer evaluations rather than $t$8 autoregressive token steps or many full-sequence diffusion steps. The method is therefore parallel within levels but sequential across representational scales (Kumbong et al., 9 Oct 2026).

Ranking and iterative focus prediction

Multi-Location Software Model Completion applies NCM-like prediction to software-model evolution. Given an already changed anchor node, NextFocus scores every other model node using global embeddings and an attention-based pair classifier. The output is a ranked set of additional focus nodes:

$t$9

The predicted locations are passed to a local completion system, after which the model can be updated and another focus prediction performed. This is not conventional token-style autoregressive NCM; it is evolution-aware, pairwise, set-oriented, and ranking-based (Welter et al., 20 Jan 2026).

4. Objectives, training, and inference

NCM objectives vary according to whether concepts are discrete, continuous, multi-label, hierarchical, or ranked.

For single next-concept prediction, sigmoid outputs support multi-label targets:

$A_t$0

If concepts are mutually exclusive, a softmax objective may be used. For next-concept sets, all concepts in $A_t$1 are positive labels, analogous to next-basket recommendation (Chen et al., 2021).

For lexical concept sets, the intended objective is to assign probability to any valid realization. A marginal form is:

$A_t$2

The formulation in “Beyond Tokens” is not fully specified for multi-token alternatives, and the paper does not establish a separate concept-level autoregressive decoder (Iyer et al., 16 Jan 2026).

For latent product-quantized concepts, ConceptLM and NCP-ArchPreview predict segment-wise code distributions and compare the expected codebook embedding with a continuous target using mean-squared error. NCP-ArchPreview uses:

$A_t$3

The VQ loss organizes the codebooks, NCP predicts future latent states, and NTP preserves fine-grained generation.

Mimir predicts continuous SONAR sentence embeddings with MSE and uses diffusion denoising during inference. It appends an “End of text.” sentence whose embedding acts as a semantic stopping marker. Generation stops when the cosine similarity between a predicted embedding and the stopping embedding reaches at least $A_t$4 (Musacchio et al., 24 May 2026).

Multiscale NCM uses a level-wise categorical objective over VQ codes, with optional level weighting:

$A_t$5

Conditioning corruption replaces some previously generated codes during training, reducing exposure bias when later levels must operate on imperfect earlier predictions (Kumbong et al., 9 Oct 2026).

AnDa and NextFocus illustrate two non-generative forms of NCM. AnDa uses binary cross-entropy over candidate future items or concepts. NextFocus trains a focal-weighted binary pair classifier but uses its probabilities for ranking. These systems demonstrate that NCM may predict a set or a ranked focus rather than generate a single sequential symbol.

5. Empirical evidence and applications

Recommendation and intent prediction

AnDa was evaluated on EPR, SPR, and Ta-Feng next-basket datasets. It was the best method on all three datasets, with reported HR@5, NDCG@5, and MAP values of $A_t$6, $A_t$7, and $A_t$8 on EPR; $A_t$9, $$\text{context} \rightarrow \text{next concept} \rightarrow \text{surface realization}.$$0, and $$\text{context} \rightarrow \text{next concept} \rightarrow \text{surface realization}.$$1 on SPR; and $$\text{context} \rightarrow \text{next concept} \rightarrow \text{surface realization}.$$2, $$\text{context} \rightarrow \text{next concept} \rightarrow \text{surface realization}.$$3, and $$\text{context} \rightarrow \text{next concept} \rightarrow \text{surface realization}.$$4 on Ta-Feng (Chen et al., 2021). Ablations indicate contributions from periodic embeddings, dynamic attributes, intra-basket attention, intra-attribute attention, and time-level attention.

AlipayKG reports 90% coverage of user problems and a $$\text{context} \rightarrow \text{next concept} \rightarrow \text{surface realization}.$$5 online search CTR increase. Its service–intent matching experiment reports Micro-F1 of $$\text{context} \rightarrow \text{next concept} \rightarrow \text{surface realization}.$$6 for multimodal matching with graph embeddings, compared with $$\text{context} \rightarrow \text{next concept} \rightarrow \text{surface realization}.$$7 for text-only StructBERT. These are semantic normalization and downstream-task results, not direct next-concept prediction results (He et al., 2023).

Language modeling

“Beyond Tokens” reports lower ordinary NTP perplexity, improved normalized cross-domain transfer, and improvements across several downstream classification benchmarks for concept-aware variants, while also documenting mixed individual cells and inconsistencies in perplexity tables (Iyer et al., 16 Jan 2026).

ConceptLM reports improvements over ordinary GPT and Pythia baselines across downstream benchmarks and perplexity evaluations. For example, average GPT-2 downstream accuracy increases from $$\text{context} \rightarrow \text{next concept} \rightarrow \text{surface realization}.$$8 to $$\text{context} \rightarrow \text{next concept} \rightarrow \text{surface realization}.$$9 at 160M parameters and from $\text{need}$0 to $\text{need}$1 at 1.5B parameters. Average perplexity decreases from $\text{need}$2 to $\text{need}$3 at 160M and from $\text{need}$4 to $\text{need}$5 at 1.5B (Liu et al., 9 Feb 2026).

Mimir demonstrates multilingual sentence-level continuation with a 1.6B-parameter model trained on $\text{need}$6 sentences in 46 languages. Its continuation evaluation reports average L2 errors of $\text{need}$7 on C4, $\text{need}$8 on MultiEURLEX, and $\text{need}$9 on Wiki40B. On XL-Sum, multilingual Mimir with direct task supervision obtains ROUGE-L $\text{provide}$0, while it obtains $\text{provide}$1 on MLQA. It underperforms Llama 3.1 models on the reported word-sense-disambiguation evaluations (Musacchio et al., 24 May 2026).

NCP-ArchPreview scales latent concept modeling to $\text{provide}$2B parameters and $\text{provide}$3T tokens. Relative to OLMo-3-7B, it reaches the final Stage-1 loss using $\text{provide}$4 of the tokens and improves the Stage-1 downstream macro-average by $\text{provide}$5 points. GSM8K improves from $\text{provide}$6 to $\text{provide}$7. The model also supports VQ-only domain adaptation using approximately 17M trainable parameters and improves mean accepted length in DFlash2 speculative decoding by $\text{provide}$8 (Team et al., 9 Sep 2026).

Fast generation and reasoning

Multiscale NCM reports Gen. PPL of $\text{provide}$9 on LM1B at 12 function evaluations, compared with diffusion baselines evaluated at 128 function evaluations. On XSum, it obtains ROUGE-1/2/L of $$\text{Order coffee delivery} = (\text{order coffee},\text{service type},\text{delivery}).$$0 at eight function evaluations. On Sudoku, it obtains $$\text{Order coffee delivery} = (\text{order coffee},\text{service type},\text{delivery}).$$1, $$\text{Order coffee delivery} = (\text{order coffee},\text{service type},\text{delivery}).$$2, and $$\text{Order coffee delivery} = (\text{order coffee},\text{service type},\text{delivery}).$$3 accuracy on easy, medium, and hard boards using 12 function evaluations, exceeding the listed autoregressive, diffusion, and flow baselines (Kumbong et al., 9 Oct 2026).

The Sudoku results illustrate a distinct advantage of coarse-to-fine refinement: later levels can revise globally inconsistent earlier choices. This contrasts with left-to-right autoregressive generation, which commits earlier positions before later positions are generated.

Software-model completion

NextFocus achieves an average Precision@$$\text{Order coffee delivery} = (\text{order coffee},\text{service type},\text{delivery}).$$4 of $$\text{Order coffee delivery} = (\text{order coffee},\text{service type},\text{delivery}).$$5 for $$\text{Order coffee delivery} = (\text{order coffee},\text{service type},\text{delivery}).$$6, compared with $$\text{Order coffee delivery} = (\text{order coffee},\text{service type},\text{delivery}).$$7 for semantic similarity and $$\text{Order coffee delivery} = (\text{order coffee},\text{service type},\text{delivery}).$$8 for historical co-change frequency and random selection. Cross-project performance declines to approximately $$\text{Order coffee delivery} = (\text{order coffee},\text{service type},\text{delivery}).$$9, indicating substantial dependence on project-specific evolution history (Welter et al., 20 Jan 2026).

The approach performs particularly well for recurring renaming, type replacement, identifier updates, and some additions of new domain concepts. It performs poorly for hierarchy changes, uncommon behavioral changes, loosened constraints, and some novel concepts.

Interactive concept construction

Agile Modeling reports a median labeling time of $\text{isA}$0 seconds per image and a mean time of $\text{isA}$1 minutes $\text{isA}$2 seconds $\text{isA}$3 seconds for 100 images. After approximately five minutes of user effort, the resulting model exceeds the zero-shot baseline by at least 3 percentage points in AUC PR; after approximately 24 minutes, the reported gain reaches 16% (Stretcu et al., 2023).

On difficult concepts, User-100—600 total user labels—outperforms Crowd-500 with 3,000 crowd labels. This demonstrates the importance of aligning labels with the intended user-specific concept, but it does not establish an automatic next-concept selection mechanism.

6. Limitations, interpretive issues, and research directions

A fundamental issue is concept definition. Human-defined concepts can be ambiguous, subjective, hierarchical, polysemous, or context-dependent. Automatically generated synonym and hypernym sets can contain false synonyms, inappropriate hypernyms, grammatical mismatches, and culturally biased associations (Iyer et al., 16 Jan 2026). Latent VQ concepts avoid external annotation but are not guaranteed to correspond to human-interpretable semantic units (Liu et al., 9 Feb 2026, Team et al., 9 Sep 2026).

Granularity is also unresolved. Sentence-level concepts in Mimir are broad, fixed segmentation units, while four-token latent concepts may cross linguistic boundaries. Multiscale NCM represents one sequence at several resolutions, but its levels are fixed by the tokenizer schedule rather than dynamically selected from linguistic structure (Musacchio et al., 24 May 2026, Team et al., 9 Sep 2026, Kumbong et al., 9 Oct 2026).

Temporal and structural leakage require careful control. Dynamic attributes may be missing, delayed, noisy, or observed after the behavior they are meant to explain. Historical model-evolution data may contain unrelated co-occurring changes. Chronological train-validation-test splits, timestamping, and causal masking are therefore necessary.

Computational complexity depends on the NCM form. Temporal self-attention has approximately $\text{isA}$4 complexity, while intra-set attention has $\text{isA}$5. Full-vocabulary concept scoring costs approximately $\text{isA}$6. NextFocus evaluates every candidate node in a model, and multiscale NCM reduces but does not eliminate sequential generation because levels remain ordered. Diffusion-based sentence concept prediction introduces iterative denoising cost, while coarse-to-fine NCM introduces VQ-VAE training and reconstruction constraints.

Evaluation remains heterogeneous. Recommendation systems report HR@5, NDCG@5, and MAP; ranking systems report Precision@$\text{isA}$7; language systems report perplexity, downstream accuracy, ROUGE, BLEU, or embedding-space distances. These metrics are not directly interchangeable. Mimir’s L2 and round-trip L2 do not establish token-level likelihood superiority, while concept-aware token training can improve perplexity without proving that models generate an explicit concept sequence. Attention weights also indicate model reliance rather than causal explanation.

Several research directions follow from the existing systems:

• Dynamic concept boundaries: replace fixed token spans or fixed sentence segmentation with content-dependent boundaries.
• Hierarchical and multi-resolution objectives: jointly predict specific concepts, abstractions, and their $\text{isA}$8 relations.
• Concept-to-surface realization: separate semantic prediction from lexical, visual, or structural realization.
• Cross-attention between context and concepts: identify which dynamic attribute or prior concept makes a future concept relevant.
• Time-varying concept representations: model concept drift, evolving meanings, and changing user preferences.
• Uncertainty and disagreement modeling: preserve multiple valid interpretations rather than collapsing them into one label or one embedding.
• Adaptive computation: stop coarse-to-fine generation early for easy examples and allocate additional levels to difficult ones.
• Graph-constrained prediction: use prerequisite, semantic, co-change, or ontology structure as auxiliary representations or decoding constraints.
• Robust evaluation: combine token-level likelihood, concept-set recall, hierarchical distance, semantic fidelity, calibration, domain transfer, user alignment, and computational cost.
• Long-context validation: test whether compressed concept sequences provide measurable advantages beyond the reported context lengths and datasets.

NCM is therefore best understood as a design space rather than a single model family. Its defining operation is prediction at a level above the immediate surface event: intent instead of raw interaction, semantic equivalence class instead of one word, sentence embedding instead of token sequence, latent multi-token state instead of individual tokens, global model location instead of local edit, or coarse representation instead of final detail. The strongest common hypothesis is that such targets provide broader predictive structure and can improve generation, recommendation, completion, or reasoning when integrated with an appropriate lower-level realization mechanism.