Dynamic Large Concept Models: Latent Reasoning in an Adaptive Semantic Space (2512.24617v1)

Abstract: LLMs apply uniform computation to all tokens, despite language exhibiting highly non-uniform information density. This token-uniform regime wastes capacity on locally predictable spans while under-allocating computation to semantically critical transitions. We propose $\textbf{Dynamic Large Concept Models (DLCM)}$, a hierarchical language modeling framework that learns semantic boundaries from latent representations and shifts computation from tokens to a compressed concept space where reasoning is more efficient. DLCM discovers variable-length concepts end-to-end without relying on predefined linguistic units. Hierarchical compression fundamentally changes scaling behavior. We introduce the first $\textbf{compression-aware scaling law}$, which disentangles token-level capacity, concept-level reasoning capacity, and compression ratio, enabling principled compute allocation under fixed FLOPs. To stably train this heterogeneous architecture, we further develop a $\textbf{decoupled $μ$P parametrization}$ that supports zero-shot hyperparameter transfer across widths and compression regimes. At a practical setting ($R=4$, corresponding to an average of four tokens per concept), DLCM reallocates roughly one-third of inference compute into a higher-capacity reasoning backbone, achieving a $\textbf{+2.69$\%$ average improvement}$ across 12 zero-shot benchmarks under matched inference FLOPs.

• The paper introduces a novel hierarchical framework that dynamically segments tokens into semantically coherent concepts for targeted reasoning.
• It leverages adaptive compute allocation through compression-aware scaling and μP parametrization to enhance efficiency and reduce FLOPs.
• Empirical results on zero-shot reasoning benchmarks demonstrate improved performance at concept boundaries while trading minor token-level precision.

Dynamic Large Concept Models: Hierarchical Latent Reasoning in Adaptive Semantic Space

Motivation and Background

Current mainstream autoregressive LLMs uniformly distribute compute across all tokens, regardless of semantic structure or information density. This uniformity leads to systematic inefficiency: predictable regions and high-information boundaries are treated identically, with no explicit mechanism for abstraction or adaptive compute allocation. Attempts to move reasoning above the token level, such as latent reasoning or coarse segmentation, either introduce interpretability and scalability limitations, or rely on rigid, human-imposed boundaries.

Dynamic Large Concept Models (DLCM) address these deficiencies by introducing a hierarchical framework in which variable-length, semantically coherent “concepts” are discovered end-to-end from latent representations. Language is processed through dynamic boundary detection and pooled into concepts, on which deep, computationally intensive reasoning is performed. This explicit separation of semantic content and the mechanism of reasoning enables adaptive inference aligned with local information density.

DLCM Architecture and Semantic Compression

DLCM is structured into four stages: (1) token-level encoding, (2) dynamic segmentation with boundary detection and pooling, (3) deep reasoning in compressed concept space, and (4) token-level reconstruction via causal cross-attention. Boundary detection is implemented through localized dissimilarity in learned representation space, employing temperature sharpening and Bernoulli sampling for training and hard thresholding for inference.

Tokens between boundaries are pooled and projected into a higher-dimensional concept space. The central “concept backbone” transformer then performs causal reasoning on these compressed representations; the output is used in a cross-attention-enabled decoder to reproduce token-level predictions. This approach enables most compute to be concentrated where semantic transitions and high-complexity reasoning are required, while routine token sequences are handled with minimal capacity.

Figure 2: Top: Average loss comparison between concept model (blue) and baseline model (orange) across relative positions within concepts. Bottom: Loss difference (Concept - Baseline), with green showing improvement at boundaries and red indicating degradation internally.

Analysis of DLCM's behavior demonstrates a "U-shaped" loss profile within concepts. Marked loss improvements at concept boundaries—where tokens exhibit higher entropy and require more reasoning—validate that resource allocation shifts toward semantic transitions. Mildly increased loss in concept interiors reveals the anticipated trade-off: reduced granularity and prioritization of coherent semantic representation at the expense of uniform token-level precision.

Compression-Aware Scaling Laws and Compute Allocation

Hierarchical compression fundamentally alters the scaling dynamics known from uniform LLMs. DLCM introduces a scaling law that jointly models total parameters, data, compression ratio, and the fraction of compute allocated to the concept backbone:

$$L(N, D, R, P) = E_0 + \frac{A_\text{token}}{(N(1-P))^{\delta_1}} + \frac{A_\text{concept} R^\gamma}{(NP)^{\delta_2}} + \frac{A_\text{data}}{D^\alpha}$$

where $P$ is the backbone parameter ratio, and $R$ is the token-to-concept compression factor. This law predicts full training trajectories ($R^2 > 0.98$) and late-stage decay ($R^2 = 0.93$), supporting principled architecture selection under equal-FLOPs constraints.

Figure 4: Full training trajectory fit. Loss predicted by the compression-aware scaling law vs. empirical loss across model scales, compression factors, and training steps.

Figure 6: Decay-phase fit. The law captures late-stage convergence behavior with $R^2 = 0.93$.

Hyperparameter stability is achieved through decoupled Maximal Update Parametrization ($\mu$P), allowing for zero-shot transferability of learning rates and optimizer parameters across heterogeneous model widths and compression regimes.

Figure 3: Hyperparameter tuning and transfer under $\mu$P: proxy model and large-scale settings exhibit consistent minima by appropriately scaling learning rates with component widths.

Architectural Efficiency and FLOPs Analysis

By compressing token sequences (e.g., $R=4$), DLCM shifts computation to a wider, higher-capacity reasoning backbone without proportional increases in inference cost, yielding up to 34% FLOPs savings. Efficiency scales monotonically with compression and percentage of parameters allocated to the backbone. The primary configuration ($P=60\%$, $R=4$) achieves best empirical and theoretical trade-offs.

Figure 5: (a) Loss/FLOPs efficiency across backbone proportions $P$ and compression ratios $R$. (b) FLOPs savings relative to baselines, for DLCM at $P=60\%$, $R=4$.

Practical Training Stability and Segmentation

Direct end-to-end discrete boundary learning is unstable under strong cross-entropy loss pressure; the model tends to reduce compression over time. DLCM remedies this by decoupling segmentation and language loss, and by employing a global regularization strategy to maintain stable compression ratios. Rule-based predictors, or globally-regularized learned predictors, maintain tight control over effective compression, and adaptively vary concept length with content domain.

Figure 7: Average compressed sequence length over training steps. The rule-based predictor (purple) maintains stable compression, while the learned boundary predictor (red) drifts away from target compression.

This adaptivity enables domain-aware compression: technical prose, code, and math are segmented differently, maximizing information retention within the global compression budget.

Computational Implementation and Attention Optimization

DLCM's cross-attention between tokens and compressed concepts induces irregular attention maps. Concept replication strategies—aligning key/value sequences to match token positions—enable the use of Flash Attention kernels under standard $L\times L$ causal masks. This achieves 1.26–1.73$\times$ speedup over flexible, mask-based attention, with the advantage increasing with sequence length and being insensitive to model width.

Figure 1: Flash Attention Varlen speedup vs. sequence length—higher speedup for longer sequences, consistently across hidden sizes.

Benchmark Results and Empirical Trade-offs

On 12 zero-shot reasoning benchmarks, DLCM achieves a +2.69% average improvement over a parameter-matched LLaMA-based baseline under matched FLOPs—weighted toward multi-step reasoning, hypothesis selection, and commonsense inference. Significant gains are observed on OpenBookQA (+3.00%), PIQA (+2.42%), and CommonsenseQA (+1.64%), with minor regressions on fine-grained lexical understanding tasks (e.g., BoolQ, -1.47%). This non-uniform gain profile directly reflects the architectural bias introduced by adaptive semantic compression.

Theoretical and Practical Implications

DLCM's results assert that the token-uniform computation paradigm of current LLMs is suboptimal, both in reasoning efficiency and in resource use. Hierarchical segmentation and compute reallocation enable larger effective model capacity at fixed inference costs, shifting the scaling bottleneck from total parameter count to the granularity and structure of semantic computation.

Practically, concept-based architectures are positioned to yield substantial savings in deployment cost for reasoning-intensive applications. The compression-aware scaling analysis and $\mu$P parametrization offer a theoretical framework for composable, heterogeneous neural architectures. The implementation pipeline, including boundary regularization and attention optimization, demonstrates immediate transferability to LLM production stacks.

Conclusion

Dynamic Large Concept Models introduce an architectural break from token-level uniformity by enabling adaptive reasoning in learned semantic spaces. Through hierarchical segmentation, concept-based computation, and compression-aware scaling, DLCM achieves higher reasoning competence with lower FLOPs, and exhibits a clear advantage on tasks aligned with non-uniform information density. These findings motivate future exploration of hierarchical abstraction, dynamic compute allocation, and multi-level reasoning in neural language systems, extending beyond the limits of token-level autoregression.