NCP-ArchPreview Technical Report: Moving towards Latent Space Language Models through Next Concept Prediction

Abstract: We introduce NCP-ArchPreview, a latent-space LLM that pushes autoregressive pretraining beyond standard next-token prediction (NTP). Alongside NTP, the model learns through Next Concept Prediction (NCP) to predict discrete concepts that span multiple tokens, introducing an explicit and more challenging concept-level objective while preserving standard token-level autoregressive generation. NCP-ArchPreview builds a latent space by constructing a product-quantized concept vocabulary directly from its hidden states, and subsequently learns to predict future concepts via a dedicated Concept Module. These predicted concepts are then fed back to the token level to guide subsequent generation, with NTP and NCP trained jointly end-to-end. We scale this architecture to 8.9B parameters and train it on 5.73T tokens from the Dolma-3 dataset, marking the largest demonstration of a latent-space LLM to date. Remarkably, by consuming only 51.3% of the total training tokens, NCP-ArchPreview achieves the final pretraining loss of OLMo-3-7B. Following full pretraining, it outperforms OLMo-3-7B by 2.45 points on the downstream macro-average, including a notable 5.99-point gain on GSM8K. Controlled experiments isolate a clear progression of performance gains stemming from both the latent architecture and the NCP objective. Furthermore, utilizing only 85% of the standard computation, NCP-ArchPreview approaches the training loss of a strictly parameter-aligned 8.9B baseline. The learned latent space remains highly valuable after the pretraining stage: updating just the 17M-parameter VQ module yields a novel, lightweight interface for domain adaptation, while a simple injection of concept representations into a DFlash2 drafter improves the mean accepted length by 4.17% with negligible overhead.

• The paper presents NCP-ArchPreview, an 8.94B-parameter latent-space language model that augments next-token prediction (NTP) with next-concept prediction (NCP), showing up to 1.95× token-efficiency improvement and lower Stage-1 loss compared to OLMo-3-7B.
• The model achieves a final Stage-1 loss 0.091 lower than OLMo-3-7B, and a 1.74× compute-efficiency improvement from scaling-law experiments.
• The FBS architecture of NCP-ArchPreview involves hierarchical residual routing, a Concept Module, and joint training objectives, which collectively improve both optimization efficiency and downstream performance.

The paper presents NCP-ArchPreview, an 8.94B-parameter latent-space LLM that augments next-token prediction (NTP) with next-concept prediction (NCP). Its central claim is that a LLM can be trained to predict structured, discrete latent representations spanning multiple tokens while retaining ordinary autoregressive token generation. The model is pretrained for 5.73T tokens from Dolma-3 and is evaluated against OLMo-3-7B under matched data, parameter, and computation settings. The report also studies scaling behavior, numerical stability, lightweight adaptation, multi-token prediction, and speculative decoding (2609.10715).

Motivation and contribution

Standard NTP supplies supervision at the granularity of individual surface tokens. Although Transformer hidden states often encode longer-range semantic structure, that structure is not directly specified by the training objective. NCP-ArchPreview introduces an explicit prediction target above the token level: contiguous groups of four token representations are compressed into concept representations, quantized into a learned discrete latent vocabulary, and predicted autoregressively by a dedicated Concept Module.

The architectural contribution is therefore not merely an auxiliary prediction head. The predicted concepts are fed back into the token pathway and condition subsequent token generation. NTP and NCP are optimized jointly, so the latent pathway participates in both representation learning and generation. The paper argues that this arrangement improves optimization efficiency while preserving the input-output behavior of a conventional causal LLM.

The principal empirical claims are substantial. Relative to OLMo-3-7B trained on the same data, NCP-ArchPreview:

• reaches OLMo-3-7B’s final Stage-1 loss after only 51.3% of the training tokens, corresponding to a reported $1.95\times$ token-efficiency improvement;
• achieves a final Stage-1 loss lower by 0.091;
• improves the Stage-1 downstream macro-average by 2.45 percentage points;
• improves GSM8K by 5.99 points;
• approaches a parameter-aligned vanilla Transformer while using 85% of its analytical training computation;
• obtains a reported $1.74\times$ compute-efficiency improvement from scaling-law experiments.

These comparisons are strengthened by progressive ablations, although some of the efficiency claims depend on analytical FLOPs rather than end-to-end wall-clock measurements.

Architecture and latent representation

NCP-ArchPreview partitions the token-level computation into a 16-layer Token Encoder and a 16-layer Token Decoder, with an 8-layer Concept Module between them. The Token Encoder produces hidden states at the original token resolution. Mean pooling over groups of four states produces a concept sequence one quarter as long as the token sequence. The Concept Module operates on this compressed sequence, while the Token Decoder returns to token resolution.

The latent vocabulary is constructed directly from Token Encoder hidden states. Each continuous concept vector is divided into 32 segments, and each segment is quantized against a codebook of 128 entries with dimension 128. Product quantization therefore provides a nominal latent vocabulary capacity of $128^{32}$ combinations without requiring a monolithic codebook of that size. The codebooks are trained using a vector-quantization loss with stop-gradient targets, allowing the codebook entries to track the latent distribution without directly forcing the Token Encoder toward the quantization objective.

The Concept Module predicts a probability distribution over codewords independently for each product-quantization segment. Rather than selecting a discrete codeword through an argmax or sampling operation, it computes the expectation of each codebook under the predicted distribution. This produces a differentiable predicted concept constrained to the span of the learned codebooks. The NCP objective minimizes the distance between this predicted concept and the detached continuous concept target.

The resulting predicted concepts are repeated to token resolution and causally shifted before being added to the Token Encoder states. The shift prevents the model from using a concept derived from tokens whose prediction is being supervised. The fused representation is then processed by the Token Decoder under the standard causal NTP objective.

Figure 1: NCP-ArchPreview combines token-level encoding and decoding with compressed concept prediction, product-quantized latent representations, and hierarchical residual routing.

This construction distinguishes NCP from multi-token prediction. MTP predicts several future surface tokens, whereas NCP predicts a latent representation summarizing a multi-token span. The target is thus not simply a set of additional token positions but a learned, structured representation intended to capture dependencies at a coarser temporal and semantic granularity.

Hierarchical residual routing

The three modules operate at different depths and sequence resolutions, creating a potential information bottleneck between token and concept streams. NCP-ArchPreview addresses this with two forms of dynamic residual routing.

Intra-Module Residual Connections (IRC) allow each module to combine representations from multiple preceding layers. A lightweight MLP produces input-dependent coefficients over candidate states, generalizing the fixed residual addition of a standard Transformer. The initialization selects the most recent residual-updated state, so the model begins close to a conventional residual architecture.

Cross-Module Residual Connections (CRC) transfer representations between the Token Encoder, Concept Module, and Token Decoder. Source states are aligned by pooling or repetition when sequence granularities differ. Target-conditioned coefficients determine how information from different source depths is mixed, followed by a learned diagonal scaling. The three principal routes are Token Encoder to Concept Module, Token Encoder to Token Decoder, and Concept Module to Token Decoder.

The residual ablation supports a cumulative interpretation of these mechanisms. On a 1B-scale model after 150B tokens, IRC reduced the loss by 0.0273 at an additional analytical cost of 0.024%. IRC combined with CRC reduced it by 0.0323 at 0.051% additional analytical FLOPs. A Block AttnRes alternative reduced the loss by only 0.0180 under a comparable analytical overhead. The result indicates that depth-wise and cross-module routing are useful, but the reported FLOP accounting excludes source-state materialization, memory traffic, reductions, and kernel-launch overhead. Consequently, the computational cost of the complete mechanism may be higher in practice than the analytical comparison suggests.

Joint training objectives

Training minimizes a weighted sum of three objectives: NTP, NCP, and VQ fitting.

NTP provides dense token-level supervision and preserves standard autoregressive generation. NCP trains the Concept Module to predict the next pooled concept from preceding concepts and also propagates gradients into the Token Encoder through the concept history. The target concept is detached, preventing the model from trivially moving the target representation to reduce the prediction error. The VQ loss updates the codebooks toward the continuous concepts while leaving the Token Encoder unaffected by that loss.

This gradient structure is important. The Token Encoder is encouraged by NCP to produce states that are predictive of future concept representations, but the codebooks are treated as a representation of the existing latent distribution rather than as a direct constraint on the encoder. The model therefore combines a learned target space with end-to-end predictive pressure.

Pretraining efficiency and downstream results

The main comparison uses the OLMo-3 curriculum: Dolma 3 Mix in Stage 1 and Dolma 3 Dolmino in Stage 2. NCP-ArchPreview preserves OLMo-3’s principal dimensions, including hidden size 4,096, FFN size 11,008, 32 attention heads, and an 8,192-token context length. The additional Concept Module and routing machinery produce a model of approximately 8.94B parameters.

The training curves show a persistent token-level loss advantage despite identical training data. During Stage 1, NCP-ArchPreview converges to the final OLMo-3-7B loss after 51.3% of the tokens and finishes 0.091 lower. During Stage 2, it matches the OLMo-3-7B endpoint after 66.2% of the tokens and finishes 0.027 lower, corresponding to a reported $1.51\times$ convergence speedup.

Figure 2: NCP-ArchPreview maintains a lower loss throughout Stage 1 and reaches the OLMo-3-7B endpoint using approximately half as many tokens.

The downstream results broadly follow the pretraining-loss advantage in Stage 1. NCP-ArchPreview’s overall macro-average is 49.04 compared with 46.59 for OLMo-3-7B. The largest reported gains include GSM8K, which rises from 39.27 to 45.26, and PiQA, which rises from 72.25 to 80.85. The MATH aggregate improves from 20.79 to 24.54, while the code aggregate increases from 25.15 to 27.79.

The Stage-2 advantage is smaller: the overall macro-average increases from 56.98 to 57.57. The code aggregate declines by 0.65 points, with HumanEval decreasing by 3.69 points and BigCodeBench decreasing by 1.16 points, despite improvements on several other code benchmarks. This result qualifies the paper’s broader claim that lower language-modeling loss consistently translates into downstream capability. The authors observe that Stage-2 data contain only approximately 10% code, so aggregate optimization can improve while underrepresented capabilities weaken.

The paper’s own checkpoint analysis reports a clearer correspondence between decreasing loss and downstream performance during Stage 1. In Stage 2, however, recipes with lower final training loss can produce worse downstream results, indicating that the relationship depends on data distribution and task domain. This distinction is methodologically important: the architecture demonstrates a robust optimization benefit, but the capability benefit is not invariant across training stages.

Ablations and scaling behavior

The progressive ablation separates the effects of the Concept Module, hierarchical residual routing, and NCP. Adding the Concept Module to the vanilla OLMo-3 configuration improves loss. Adding residual routing produces a further improvement, and adding the NCP objective yields another reduction. The ordering supports the interpretation that both the latent architecture and the explicit concept objective contribute independently.

Figure 3: Progressive addition of the Concept Module, hierarchical residual routing, and NCP produces successive training-loss improvements.

The matched comparisons address a major confound: NCP-ArchPreview contains more parameters than the 7B baseline. A standard Transformer with 40 blocks is used as a parameter-aligned baseline, while a 34-block model is used as a computation-aligned baseline. Because the Concept Module processes a sequence compressed by a factor of four, its blocks add approximately standard-block parameter capacity at substantially lower analytical sequence-level computation.

NCP-ArchPreview outperforms the vanilla and computation-aligned baselines and approaches the parameter-aligned 40-block model while using only 34 block-equivalents of analytical computation, or 85% of the parameter-aligned baseline’s computation. This supports the claim that the gains are not explained solely by increased parameter count or compute, although the comparison still relies on a particular accounting of the compressed Concept Module and does not fully measure implementation overhead.

The scaling experiments search over model/data allocations and training hyperparameters at several fixed FLOPs budgets. The fitted curves indicate that NCP-ArchPreview reaches lower validation loss than OLMo-3 at comparable budgets, yielding the reported $1.74\times$ compute-efficiency improvement.

Figure 4: IsoFLOP scaling curves indicate lower validation loss for NCP-ArchPreview across the evaluated compute budgets.

The scaling result is suggestive but should be interpreted within the experimental protocol. Each point reflects a searched allocation and selected hyperparameters, and the FLOPs estimate is based on analytical per-token computation. The result establishes an efficiency advantage under the paper’s accounting and search space, not a universal scaling law independent of implementation or training recipe.

Numerical stability under Muon

The report identifies a separate engineering issue involving the interaction between Muon-style matrix optimization and OLMo-3’s layer-wise Q/K normalization. Training can produce growing attention logits, head-wise Q/K norm outliers, and severe gradient-norm spikes.

Figure 5: Layer-wise Q/K normalization under the Muon baseline is associated with attention-logit growth, gradient spikes, and head-wise norm outliers.

Ablations show that the instability is reproduced under Muon and is mitigated when Q/K normalization is applied per attention head rather than across the layer. The proposed explanation is that full-matrix Muon updates couple multiple heads, while layer-wise normalization constrains only their aggregate scale. Individual heads can therefore develop disproportionate Q/K alignment and dominate the attention logits.

Figure 6: Per-head Q/K normalization keeps attention logits and post-normalization Q/K norms bounded in the reported stability ablation.

The main experiments nevertheless retain layer-wise normalization to preserve a controlled comparison with OLMo-3-7B. This creates an explicit tension in the report: the configuration used for the principal results is not the configuration identified as most stable. The paper does not establish whether replacing layer-wise normalization with per-head normalization would preserve, improve, or reduce the reported downstream and scaling advantages.

Lightweight adaptation through the latent interface

A notable secondary result is that the learned concept space functions as an adaptation interface. The VQ setting freezes the token-level backbone and updates only the VQ codebooks and concept-prediction heads, totaling 17M trainable parameters. Unlike LoRA, it introduces no additional parameters.

For code adaptation, VQ training increases the NCP-ArchPreview code average from 30.04 to 32.69, a 2.65-point gain, and is the only evaluated setting that improves all four reported code tasks. For mathematics, it raises the average from 30.56 to 34.83, a 4.27-point improvement, while the general benchmark average increases by 0.39. For knowledge adaptation, TriviaQA exact match rises from 40.28 to 49.47, a 9.19-point gain, while the general average changes by only 0.03 points.

The adaptation profile is domain-dependent. VQ is less effective than full training or LoRA for factual knowledge, plausibly because factual associations encoded in the backbone’s feed-forward layers cannot be directly rewritten by changing only the concept interface. Its advantage is reduced interference: for mathematics, it retains more general capability than either full training or LoRA, and for knowledge it produces nearly no change in the general average.

The computational measurements are favorable. With micro-batch size one on eight GPUs, VQ training reaches 15,632 tokens per second per GPU, compared with 10,400 for LoRA and 7,739 for full training. Its memory usage is 32.4%, compared with 49.0% for LoRA and 90.3% for full training. Thus, the latent interface offers a practical adaptation mechanism, although the results are based on a limited set of domains and do not establish that VQ adaptation is generally preferable to parameter-efficient alternatives.

Figure 7: VQ adaptation provides higher throughput and lower memory usage than LoRA and full-parameter training in the reported configuration.

Interaction with multi-token prediction and speculative decoding

The paper also evaluates whether NCP is complementary to conventional MTP. At 3B scale, NCP-ArchPreview without MTP obtains an average language-modeling loss of 2.3707, compared with 2.3805 for a residual-aligned OLMo-3 baseline. Adding MTP reduces the losses to 2.3664 and 2.3739, respectively. NCP therefore provides a larger improvement than MTP alone in this comparison, and the combination is additive.

The loss trajectories show an early crossover: a residual-aligned OLMo-3 model with MTP initially performs better than NCP-ArchPreview, but NCP-ArchPreview crosses below it near update 2,285 and remains better over the final 8,000 updates. The result suggests that the latent objective can require a different optimization trajectory and that early checkpoint comparisons may not reflect eventual ordering.

Figure 8: NCP-ArchPreview eventually outperforms the residual-aligned OLMo-3 baseline with MTP, while combining NCP and MTP yields the lowest loss.

The learned concepts are also injected into a DFlash2-style block-parallel speculative drafter. Concept conditioning adds only 0.04M parameters to a 1.1B-parameter drafter and does not require additional Target-model computation. Across GSM8K, MATH, HumanEval, and MBPP, mean accepted length increases from 5.933 to 6.180, a 4.17% macro-average gain. The largest improvement is on HumanEval at 7.59%.

This result supports the claim that concept representations contain information useful beyond ordinary token-level decoding. However, the experiment evaluates one drafter family, one proposal horizon, and a fixed distillation setup. It demonstrates compatibility with speculative decoding rather than establishing a general inference improvement for latent-space LLMs.

Proxy evaluation for mid-training recipe selection

The appendix introduces a 100M-token proxy evaluation intended to screen Stage-2 data recipes without repeatedly running the complete downstream suite. It uses expert trajectories from 63 sources and six domains, computes source-balanced likelihood statistics for free-form capabilities, and uses a task-native correct-versus-distractor margin for HellaSwag.

Among three matched-budget Stage-2 recipes, the proxy ordering is concordant with downstream ordering for code, mathematics, academic STEM, and logical reasoning. For example, the strongest recipe reduces proxy NLL relative to the weakest by 3.5% for function generation and 5.5% for competition mathematics, corresponding to improvements of 5.64 points on HumanEval and 6.52 points on MATH-500. Stage-2 fits achieve high reported $R^2$ values for HumanEval, MBPP, MATH-500, Minerva, and MMLU-STEM, while BBH has a substantially lower $R^2$ of 0.531 despite preserving the same rank ordering.

Figure 9: Source-balanced proxy likelihoods track the relative downstream ordering of the evaluated Stage-2 recipes for most capability–benchmark pairs.

The scope of this result is limited. The evidence covers three recipes and one model family, and the authors explicitly avoid causal attribution to individual data sources. The proxy is best understood as a within-setting recipe-screening tool, not a calibrated predictor of downstream performance for unseen architectures or distributions.

Limitations and open questions

The experiments use a maximum context length of 8,192 tokens and do not evaluate long-context training. The compressed concept sequence could alter the computational trade-off favorably at longer contexts, but this remains untested. In particular, it is not established whether fixed four-token grouping remains appropriate when dependencies span substantially longer and structurally heterogeneous regions.

The report also does not fully resolve the relationship between optimization efficiency and capability. The strongest downstream gains occur after Stage 1, whereas Stage 2 produces a much smaller aggregate improvement and several code regressions despite a lower loss. This leaves open whether the architecture’s principal advantage is improved representation learning, better optimization conditioning, or a data-dependent interaction between latent supervision and curriculum.

Several efficiency claims use analytical FLOPs that omit memory movement, source-state materialization, reduction operations, and kernel-launch overhead. The paper reports throughput and memory results for VQ adaptation, but it does not provide an end-to-end pretraining wall-clock comparison against all matched baselines. The practical cost of hierarchical residual routing at full scale therefore remains incompletely characterized.

Finally, the most stable Q/K normalization variant is not used for the principal results. It remains open whether per-head normalization changes the latent codebook dynamics, NCP quality, scaling curves, or downstream performance. More generally, the experiments establish that the proposed latent pathway works at 8.9B parameters and 5.73T tokens, but they do not determine how performance depends on the concept compression factor, codebook structure, number of Concept Module layers, or the weighting of NCP and VQ losses.

Conclusion

NCP-ArchPreview provides a large-scale demonstration that discrete concept prediction can be integrated into a standard autoregressive LLM without abandoning token-level generation. Its architecture combines a compressed Concept Module, product-quantized latent targets, causally shifted concept feedback, and hierarchical residual routing. Under the reported training protocol, these components yield lower loss, faster token-level convergence, improved Stage-1 downstream performance, and favorable compute-scaling behavior relative to OLMo-3 baselines (2609.10715).

The strongest evidence concerns optimization efficiency: the model reaches the OLMo-3-7B Stage-1 endpoint using 51.3% of the tokens and achieves a 0.091 lower final loss. The downstream results are positive but distribution-dependent, and the adaptation and speculative-decoding experiments show that the learned concept space can serve as an interface beyond pretraining. The remaining technical questions concern long-context behavior, faithful end-to-end efficiency accounting, stability under alternative normalization schemes, and the extent to which latent-space objectives consistently improve capability rather than only training loss.