Next-Concept Modeling: Fast, Parallel Coarse-to-Fine Text Generation

Abstract: Diffusion LLMs (DLMs) offer parallel sampling for faster generation than autoregressive models, an increasingly important advantage as inference becomes a dominant computational workload. However, DLMs degrade quickly with fewer sampling steps, process the full sequence at every step, and require many passes over the dataset to reach good quality. We introduce Next-Concept Modeling (NCM), a two-stage, coarse-to-fine framework: a multiscale residual VQ-VAE encodes text into a hierarchy of discrete codes, and a transformer generates one level at a time, predicting all codes within a level in parallel. At matched parameter scales, NCM achieves better generative perplexity than the strongest DLM baseline on LM1B while running $21\times$ faster. On OpenWebText, NCM is competitive with the strongest DLM while running $10\times$ faster; on XSum, it matches the strongest DLM's ROUGE-L while being $31\times$ faster. On the reasoning task of Sudoku, NCM outperforms autoregressive and diffusion baselines across all difficulty levels, using fewer function evaluations. On the reasoning task of Sudoku, NCM outperforms autoregressive and diffusion baselines across all difficulty levels, using fewer function evaluations.

• The paper introduces Next-Concept Modeling (NCM), which significantly speeds up text generation by using a learned hierarchy of discrete latent codes, reducing the need for repeated full-sequence evaluations. This approach maintains high reconstruction accuracy across multiple datasets, such as 99.99% on OpenWebText and 99.86% on LM1B.
• NCM’s coarese to fine generation achieves competitive quality with substantial efficiency gains, such as 11 times fewer evaluations and 21 times faster processing compared to diffusion baselines on LM1B, demonstrating its effectiveness in balancing generation quality and computational cost.
• Coarse-to-fine parallelism in NCM is effective for globally constrained tasks like Sudoku, achieving 99.5% accuracy on easy puzzles with fewer evaluations than competing models, indicating the model's potential for diverse and structured text generation tasks.

Problem formulation and central contribution

“Next-Concept Modeling: Fast, Parallel Coarse-to-Fine Text Generation” (2610.14162) addresses a specific limitation of diffusion LLMs (DLMs): although DLMs generate multiple tokens per iteration, they generally require repeated full-sequence evaluations, and their quality degrades sharply when the number of denoising steps is reduced. The paper argues that reducing inference cost requires changing not only the sampler but also the representation and generation schedule.

NCM replaces token-space iterative refinement with generation in a learned hierarchy of discrete latent codes. A multiscale residual VQ-VAE first maps a length-$L$ token sequence to $K$ code levels with increasing resolutions,

$\ell_1 < \ell_2 < \cdots < \ell_K=L.$

The generator then predicts level $k$ conditioned on the cumulative reconstruction of levels $1$ through $k-1$. All $\ell_k$ codes within a level are predicted in parallel. Since $K \ll L$, generation requires substantially fewer sequential model evaluations, while the early evaluations operate on short coarse representations.

The paper’s principal claim is therefore architectural rather than merely algorithmic: parallel generation becomes more efficient when the model learns the order in which information should be revealed. NCM combines representation compression, coarse-to-fine prediction, and level-specific computation into a single inference procedure.

Multiscale residual tokenization

The tokenizer is a residual VQ-VAE with progressively longer code sequences. At level $k$, the residual left by earlier levels is downsampled, quantized using a level-specific codebook, and mapped back to the full latent resolution. The cumulative reconstruction is

$\hat z^{(k)} = \hat z^{(k-1)}+ R_k(e_k(r_k)),$

where $K$0 expands the level-$K$1 code contribution to the full latent grid. The residual formulation makes later levels responsible for correcting the approximation produced by earlier levels.

A key training mechanism is nested dropout. During tokenizer training, only a prefix of the hierarchy is retained for reconstruction, forcing early levels to encode information that remains useful without access to the finer codes. This is essential to NCM’s interpretation of coarse levels as carriers of global structure rather than merely as arbitrary intermediate variables.

The tokenizer achieves extremely high full-resolution reconstruction accuracy. Exact token recovery is reported at $K$2 on OpenWebText, $K$3 on LM1B, $K$4 on TinyStories, $K$5 on XSum, and $K$6 on WMT14. Sudoku reaches $K$7 reconstruction accuracy. These results establish that the latent hierarchy can preserve the information required for exact symbolic outputs, although they do not imply that generated latent codes will be equally accurate.

Codebook utilization ranges from $K$8 on WMT14 to $K$9 on TinyStories. Thus, the tokenizer does not collapse to a small subset of codes, but utilization is uneven across tasks. The lower utilization on translation indicates that the learned representation is affected by the task distribution and target-domain structure.

Figure 1: Exact reconstruction accuracy and codebook utilization as the hierarchy is refined.

The hierarchy probes support the intended semantic interpretation. Fixing early codes while resampling later codes tends to preserve broad narrative structure while changing names, objects, and wording. Conversely, perturbing a small number of early codes can change entities and events while preserving portions of the original narrative. These observations are qualitative, but they provide evidence that the level ordering corresponds to a meaningful information hierarchy rather than only to spatial or positional compression.

NCM generation and training objective

With the tokenizer frozen, the NCM transformer predicts all code levels in a single training forward pass. At inference, it proceeds sequentially:

1. initialize the cumulative reconstruction;
2. resize and project it to the current level;
3. predict all codes at that level in parallel;
4. add the decoded level contribution to the cumulative reconstruction;
5. continue to the next level;
6. decode the completed reconstruction into tokens.

The generator factorizes the hierarchy as

$\ell_1 < \ell_2 < \cdots < \ell_K=L.$0

and further factorizes each level across positions. This conditional independence is a deliberate approximation: within-level dependencies are not modeled autoregressively. NCM instead relies on the cumulative reconstruction and the transformer’s shared contextual computation to make parallel predictions coherent.

For unconditional generation, block-diagonal attention is used. Each level’s input block already contains a cumulative reconstruction of preceding levels, so explicit attention to earlier blocks is unnecessary. This design reduces computation and avoids repeatedly processing the full hierarchy. In the TinyStories ablation, block-diagonal attention obtains a generative perplexity of $\ell_1 < \ell_2 < \cdots < \ell_K=L.$1 and sampling time of $\ell_1 < \ell_2 < \cdots < \ell_K=L.$2 seconds, compared with $\ell_1 < \ell_2 < \cdots < \ell_K=L.$3 and $\ell_1 < \ell_2 < \cdots < \ell_K=L.$4 seconds for a two-level sliding window, and $\ell_1 < \ell_2 < \cdots < \ell_K=L.$5 and $\ell_1 < \ell_2 < \cdots < \ell_K=L.$6 seconds for block-causal attention.

Figure 2: NCM’s block-diagonal attention restricts attention to the current level because earlier levels are already summarized in its input.

Training also addresses exposure bias between teacher-forced conditioning and inference-time conditioning. The model replaces a fraction of conditioning codes with other codes from the same codebook while retaining the original codes as targets. Moderate corruption improves generation, whereas excessive corruption removes too much conditioning information. This mechanism is important because errors introduced at one level can otherwise propagate through all subsequent levels.

The weighting of level losses controls the trade-off between coarse and fine supervision. Uniform code-position weighting corresponds to the natural token-level objective, while equal-level weighting gives substantially greater influence to the short coarse levels. On TinyStories, increasing the weighting exponent from $\ell_1 < \ell_2 < \cdots < \ell_K=L.$7 to $\ell_1 < \ell_2 < \cdots < \ell_K=L.$8 raises the coarse levels’ loss share from $\ell_1 < \ell_2 < \cdots < \ell_K=L.$9 to $k$0, but does not improve final generation: generative perplexity remains approximately $k$1–$k$2 for exponents up to $k$3 and worsens to $k$4 at $k$5. This result challenges the assumption that explicitly optimizing coarse-level predictability necessarily improves end-to-end generation.

Unconditional generation

The unconditional experiments evaluate TinyStories, LM1B, and OpenWebText using generative perplexity and mean per-sample unigram entropy. Reporting entropy is important because a model can improve generative perplexity by producing less diverse samples. NCM generally maintains entropy close to that of the data while using substantially fewer evaluations.

At the strongest reported settings, NCM uses 12 function evaluations on TinyStories and LM1B, and 32 on OpenWebText. The principal comparisons use 128 or 1,024 evaluations for diffusion baselines, depending on the dataset. On LM1B, NCM obtains a generative perplexity of $k$6 at 12 evaluations, compared with $k$7 for DUO at 128 evaluations. On OpenWebText, NCM obtains $k$8 at 32 evaluations, compared with $k$9 for DUO at 1,024 evaluations. NCM is therefore not uniformly superior in absolute perplexity, but its quality-cost frontier is substantially better.

Under four additional reference models, NCM obtains the best average LM1B generative perplexity: $1$0, compared with $1$1 for DUO, while using $1$2 rather than $1$3 sampling steps and approximately $1$4 billion rather than $1$5 billion training-token exposures. This robustness reduces the concern that the principal comparison is an artifact of GPT-2-large scoring.

Figure 3: NCM reaches competitive unconditional generation quality with substantially fewer function evaluations and lower sampling cost.

The paper’s strongest efficiency result is measured end to end rather than by function evaluations alone. On LM1B, NCM uses $1$6 fewer evaluations than DUO and runs $1$7 faster at better generative perplexity. On OpenWebText, it remains competitive while processing short early-level sequences, which lowers the average cost per evaluation. The result follows from two independent savings: fewer sequential levels and less computation in the coarse stages.

The level-count ablation shows that increasing the number of levels improves generative perplexity, although with diminishing returns. All tested hierarchies reconstruct at least $1$8 of tokens, so the generation-quality improvement is not explained solely by tokenizer reconstruction accuracy. It reflects a more favorable division of the prediction problem across levels. The trade-off is that increasing the number of levels also increases sequential depth, making the optimal hierarchy a quality-latency choice rather than a monotonic design parameter.

Figure 4: Increasing hierarchy depth improves generation quality with diminishing returns while preserving near-perfect tokenizer reconstruction.

The level schedule also matters. Six schedules with the same number of levels and identical endpoints produce different convergence and final perplexity. This indicates that the distribution of computation across resolutions affects optimization, not merely the total number of codes. The paper does not identify a universally optimal schedule; the selected schedules remain dataset- and implementation-dependent.

Figure 5: Alternative code-length schedules produce different training dynamics, final perplexities, and information allocations across levels.

Conditional generation and test-time computation

NCM supports conditional generation using a prefix-LM attention pattern. The source sequence is encoded by the frozen VQ-VAE embedding system, while target levels attend to the source prefix and their own current block. On WMT14 German–English, NCM reaches BLEU $1$9 with 11 evaluations without guidance. This trails the strongest diffusion baseline, E2D2, but uses $k-1$0 fewer evaluations.

On XSum, NCM obtains ROUGE-1/2/L scores of $k-1$1 with only 8 evaluations. These scores exceed the strongest unguided diffusion baseline, AURORA-LM, which obtains $k-1$2 with 128 evaluations. NCM consequently achieves a particularly strong quality-efficiency result for summarization.

Classifier-free guidance improves translation and summarization modestly. With guidance, NCM obtains BLEU $k-1$3 and XSum ROUGE-L $k-1$4 at 22 and 16 evaluations, respectively. The guidance optimum is task-dependent: ROUGE scores peak at guidance weight $k-1$5, whereas BLEU peaks at $k-1$6.

The low sampling cost also enables test-time scaling through multiple candidates and MBR selection. With 12 candidates on WMT14, NCM reaches BLEU $k-1$7 using 132 total evaluations. With 16 XSum candidates, it reaches ROUGE-1/2/L of $k-1$8 using 128 evaluations. The XSum ROUGE-L score is approximately $k-1$9 higher than the strongest diffusion baseline under a comparable sampling-time budget. This is a consequential result: NCM’s primary advantage is not only faster single-sample decoding, but also the ability to spend the saved inference budget on candidate generation and verification.

Figure 6: MBR quality increases with the number of sampled candidates, allowing NCM to convert low per-sample cost into additional test-time computation.

Sudoku reasoning

The Sudoku experiment tests whether coarse-to-fine parallelism is compatible with globally constrained generation. Unlike ordinary text metrics, Sudoku requires every output digit to satisfy row, column, and subgrid constraints simultaneously. The evaluation reports the fraction of completely solved boards.

NCM uses 12 evaluations and obtains accuracies of $\ell_k$0, $\ell_k$1, and $\ell_k$2 on easy, medium, and hard puzzles. The strongest cited baseline, FMLM+, uses 16 evaluations and obtains $\ell_k$3, $\ell_k$4, and $\ell_k$5. NCM also outperforms all autoregressive and diffusion baselines at their reported settings.

These results support the claim that a representation capable of revising a global partial solution is useful for constraint satisfaction. They do not establish that NCM performs symbolic reasoning in a general sense: Sudoku is a narrow structured task, the tokenizer is separately trained, and the comparison uses task-specific models. Nevertheless, the experiment demonstrates that parallel latent refinement need not be restricted to fluent surface-text generation.

Scaling and training efficiency

The model-scaling experiment trains LM1B generators with 97M, 317M, and 725M parameters while holding the tokenizer fixed and keeping the generator training-token budget constant. Generative perplexity improves as model size increases, while sample entropy remains high. This suggests that the hierarchical representation does not prevent conventional scaling behavior within the tested range.

Figure 7: Larger NCM generators improve LM1B generative perplexity while preserving sample diversity at a fixed training-token budget.

The training-efficiency comparison is also substantial. On LM1B, tokenizer and generator training together expose the model to approximately $\ell_k$6 billion token positions, compared with $\ell_k$7 billion for the diffusion baselines, a $\ell_k$8 reduction. On OpenWebText, NCM uses approximately $\ell_k$9 billion token positions, roughly $K \ll L$0 fewer than the more than $K \ll L$1 billion positions used by MDLM and DUO.

These figures measure token exposure rather than FLOPs. They therefore do not fully capture the computational cost of the VQ-VAE, optimizer state, sequence lengths, or architectural differences. The paper’s explanation—that a fixed coarse-to-fine ordering is easier to learn than the many denoising orders implicit in diffusion training—is plausible but not directly isolated experimentally.

Limitations and open questions

NCM depends on a separately trained tokenizer and freezes it before generator training. This decoupling simplifies optimization but prevents the representation from adapting to the generator’s modeling objective. The deterministic VQ assignment also yields a point-mass latent posterior, so the ELBO interpretation is limited: the bound is tight only when the posterior assigns essentially all probability to the tokenizer’s selected hierarchy.

The hierarchy depth and level lengths are fixed for every example. Consequently, even short or simple outputs traverse the full schedule, and the model cannot allocate fewer levels to easy instances or more computation to difficult ones. An input-dependent hierarchy is a specific unresolved design question raised by the paper.

Tokenizer errors remain consequential despite high aggregate reconstruction accuracy. The reported failures include named entities, numerals, diacritics, markup, usernames, and word boundaries. Such errors are rare on average but can invalidate exact outputs or alter meaning. Generated coarse-level errors can also propagate through later conditioning, although code corruption partially mitigates this problem.

The experiments are concentrated around approximately $K \ll L$2 million-parameter generators, with scaling tested only up to $K \ll L$3 million parameters on LM1B. The efficiency claims are therefore strongest for the evaluated model and sequence-length regimes. In addition, several conditional baseline quality results are inherited from prior work, while some timing comparisons use randomly initialized baseline weights because released checkpoints were unavailable. These protocol details limit the strength of absolute wall-clock comparisons.

Finally, intermediate coarse-to-fine outputs are not guaranteed to improve monotonically. The translation trajectory contains levels whose decoded BLEU decreases before recovering at later levels. Thus, coarse representations should be interpreted as latent planning states, not necessarily as progressively better textual outputs.

Conclusion

NCM proposes a coherent alternative to both autoregressive decoding and full-sequence diffusion refinement. Its central mechanism is a learned multiscale residual representation whose levels are generated sequentially but whose within-level codes are predicted in parallel. Across unconditional generation, summarization, translation, and Sudoku, the method combines competitive quality with substantially fewer function evaluations, $K \ll L$4–$K \ll L$5 lower reported sampling time in key comparisons, and $K \ll L$6–$K \ll L$7 lower training-token exposure than strong diffusion baselines.

The evidence supports the paper’s narrower claim that generation efficiency can be improved by learning a coarse-to-fine computation schedule in latent space. The unresolved questions concern how such hierarchies should be learned end to end, adapted dynamically to individual examples, and evaluated at larger model and dataset scales.