Scaling Embeddings Outperforms Scaling Experts in Language Models

Abstract: While Mixture-of-Experts (MoE) architectures have become the standard for sparsity scaling in LLMs, they increasingly face diminishing returns and system-level bottlenecks. In this work, we explore embedding scaling as a potent, orthogonal dimension for scaling sparsity. Through a comprehensive analysis and experiments, we identify specific regimes where embedding scaling achieves a superior Pareto frontier compared to expert scaling. We systematically characterize the critical architectural factors governing this efficacy -- ranging from parameter budgeting to the interplay with model width and depth. Moreover, by integrating tailored system optimizations and speculative decoding, we effectively convert this sparsity into tangible inference speedups. Guided by these insights, we introduce LongCat-Flash-Lite, a 68.5B parameter model with ~3B activated trained from scratch. Despite allocating over 30B parameters to embeddings, LongCat-Flash-Lite not only surpasses parameter-equivalent MoE baselines but also exhibits exceptional competitiveness against existing models of comparable scale, particularly in agentic and coding domains.

• The paper demonstrates that scaling N-gram embeddings yields a superior Pareto frontier compared to expert-based scaling in large language models.
• It introduces an efficient N-gram Embedding mechanism with sub-table decomposition and linear projection to mitigate hash collisions and maintain parameter efficiency.
• Empirical results on the LongCat-Flash-Lite model reveal practical gains in training loss and inference speed, emphasizing the benefits of embedding scaling over traditional MoE techniques.

Scaling Embeddings Outperforms Scaling Experts in LLMs: An Expert Overview

Introduction

"Scaling Embeddings Outperforms Scaling Experts in LLMs" (2601.21204) systematically addresses the limitations of Mixture-of-Experts (MoE) architectures at large scales in LLMs, highlighting the emergence of diminishing returns and increasing system bottlenecks as key challenges. The authors propose embedding scaling—especially via N-gram Embedding (NE)—as an orthogonal and substantially more efficient parameter scaling modality for LLMs. Their work evaluates comparative scaling trajectories, details architectural and systems implications, and introduces the LongCat-Flash-Lite model to empirically substantiate their claims.

N-gram Embedding: Design and Mechanism

The paper advances N-gram Embedding (NE) as the foundation for embedding scaling, leveraging multi-gram representations with vocabulary-free lookup tables. The augmented embedding for token $t_i$ is computed by summing over n-gram lookup outputs (up to order $N$), hashed and projected, yielding an expressive context-enriched token vector. To further reduce hash collisions and increase representational diversity, the NE mechanism employs sub-table decomposition ($K$ sub-tables per n-gram order) and linear projections into the embedding space, keeping parameter growth invariant with respect to $N$ and $K$.

Figure 1: Architecture of an N-gram Embedding layer highlighting augmentation of base token embeddings with hashed n-gram vectors.

Comparative Scaling: Embedding Versus Experts

Through extensive pre-training experiments on the LongCat-Flash model family, the study benchmarks the loss dynamics of NE scaling against expert scaling under equivalent parameter budgets. The findings indicate that NE-based scaling achieves a superior Pareto frontier once expert count surpasses a saturation threshold ("sweet spot"). Specifically, while expanding expert parameters in low-ratio regimes offers sharper loss reduction, the scaling efficiency plateaus and NE scaling begins to provide materially improved efficiency beyond this regime.

Figure 2: Scaling curve comparison between MoE and NE models as parameter ratio increases, with log-linear loss trends and intersection points indicating optimal NE integration timing.

Notably, the authors show that allocating more than 50% of total parameters to NE yields diminishing returns, with MoE baselines overtaking NE-enhanced models under excessive embedding budgets. This supports a regime-specific budgeting guideline (embed ≤ 50% of total params).

Architectural and Hyperparameter Sensitivities

Hash collision analysis for NE reveals critical vocabulary sizing effects, especially for 2-grams, where collision rates spike at integer multiples of the base vocabulary size. Selection of vocabulary size must thus deviate from integer multiples to prevent semantic collapse. Hyperparameter sweeps over n-gram order ($N$) and sub-table count ($K$) show that optimal performance emerges at $N=3$–$5$ and $K\geq 2$, with little sensitivity beyond these values.

Figure 3: Influence of n-gram order $N$ and sub-table count $K$ on training and validation loss—performance largely stabilizes for $N \geq 3$, $K \geq 2$.

Embedding signal amplification—via scaling factors or LayerNorm—proves essential to prevent suppression of NE contributions in residual streams, yielding a ~0.02 consistent loss reduction relative to vanilla initialization.

Figure 4: Layer-wise L2 norm analysis showing the attenuation of embedding outputs in the residual path compared to attention outputs, motivating the need for embedding amplification.

Scaling Laws Across Model Width and Depth

Scaling experiments demonstrate that increased model width (activation size) widens the performance gap in favor of NE over MoE, whereas increased depth contracts this advantage, with diminishing returns observed in models exceeding 20 layers. The window for embedding scaling effectiveness is thus maximized in wide, shallow architectures.

Efficient Inference: System Perspective

NE shifts parameter capacity from MoE to the embedding layer, dramatically reducing activated MoE parameters and alleviating bandwidth bottlenecks in decoding for long sequences. The computational cost of NE is contained by its $O(1)$ lookup property. The adoption of speculative decoding further synergizes with large batch sizes—essential to realizing throughput gains from NE—while a dedicated N-gram Cache and custom kernels minimize device synchronization and lookup overhead.

Figure 5: Training loss curves comparing LongCat-Flash-Lite (with NE) to LongCat-Flash-Lite-Vanilla (expert scaling only), highlighting consistent performance superiority tied to NE scaling.

The paper also discusses leveraging NE for ultra-fast draft modeling and early rejection in speculative decoding, enabling additional practical latency reductions.

Per-Layer Embedding Integration

The empirical comparison of PLE (Per-Layer Embedding) and PLNE (Per-Layer N-gram Embedding) to NE variants indicates that N-gram-based methods outperform standard PLE, with PLNE offering only marginal improvements and increased activation cost. The authors recommend preference for NE as the principal embedding scaling modality pending further exploration of hybrid allocation strategies.

Figure 6: Loss comparison for NE, PLE, and PLNE models; NE displays lower loss at equivalent parameter counts.

LongCat-Flash-Lite: Model Instantiation and Evaluation

LongCat-Flash-Lite—a 68.5B parameter MoE+NE model (31.4B parameters in NE, 46% of total)—is presented as a practical instantiation of the embedding scaling paradigm. Trained from scratch with extended sequence lengths and a rigorous data recipe, it outperforms parameter-equivalent MoE baselines (LongCat-Flash-Lite-Vanilla) in both training loss and downstream task accuracy.

On agentic tool use, agentic coding, general domain understanding, and mathematical reasoning benchmarks, LongCat-Flash-Lite exhibits competitive or state-of-the-art performance per activated parameter against notable contemporaries (Qwen3-Next-80B, Gemini 2.5 Flash-Lite, Kimi-Linear-48B). Notably, NE scaling delivers marked advantages in agentic and coding metrics.

System Optimizations and Fast Inference

LongCat-Flash-Lite is served via Eagle3 with advanced inference strategies: three-step speculative decoding, wide expert parallelism, single batch overlap, and kernel fusion, with PDL for SM utilization. Kernel-launch bottlenecks are addressed with operator fusion and attention combine optimizations to maintain GPU occupancy and minimize end-to-end latency.

Implications and Future Directions

The work establishes embedding scaling (especially NE) as a superior, system-efficient route to parameter expansion, outperforming expert scaling in both practical throughput and efficiency in regime-appropriate architectures. The findings suggest that the optimal scaling mix for future LLMs will increasingly rely on dynamic, context-aware allocation between embedding and expert parameters, with NE-style embedding mechanisms expected to become foundational components of high-capacity, low-latency LLM architectures.

Application implications include improved performance in agentic/coding domains, enhanced inference speed in long-context settings, and system-level reductions in memory bandwidth overhead. Theoretically, the results invite further investigation into scaling laws, embedding-parameter interactions, and architectural balances between width and depth. Future research directions include more granular layerwise allocation of NE parameters, draft modeling with NE, and optimization of caching/scheduling strategies for heterogeneous deployment scenarios.

Conclusion

Scaling LLMs by expanding embedding parameters—via N-gram Embedding—offers a robust alternative to expert-based scaling, delivering superior loss reduction, lower resource consumption, and greater throughput in both training and inference. LongCat-Flash-Lite exemplifies the practical and theoretical advances of this approach, underscoring embedding scaling's emergent role in the design of next-generation LLMs.