Layer-wise MoE Routing Locality under Shared-Prefix Code Generation: Token-Identity Decomposition and Compile-Equivalent Fork Redundancy

Abstract: In LLM-based code generation, multiple code candidates are often generated in parallel from the same prompt -- for example, in best-of-N sampling or multi-candidate code completion. These requests can share KV caches through a common prefix, yet the extent to which their Mixture-of-Experts (MoE) expert routing overlaps, and how this overlap varies across layers, remains insufficiently understood. We study Qwen3.5-35B-A3B-FP8 (256 routed experts, top-8) by performing tree-search-based branching generation from a shared prefix (851 completed codes, temperature 0.7) and analyzing the results with a compiler-output-based alignment (gcc -S -O0 assembly) that controls for token-identity confounds. Our findings are threefold: (1) At positions where both sequences generated the same token, Jaccard similarity reaches 0.649 (40x random), while even at positions with different tokens it remains 0.175 (11x random). (2) A layer-wise decomposition reveals a crossing pattern: same-token routing similarity exceeds different-token similarity across all layers, but dips in the middle layers (L14-20), while different-token similarity peaks in the middle layers at 14x random. (3) In tree-search code generation, 67% of successfully compiled codes concentrate in the top three assembly-equivalent groups, and 99.6% of within-group differences consist of comments and blank lines. We show that diversity in top-P search, including beam search, poses a significant challenge. These results refine the "context-independent routing" claim of prior work through layer-wise decomposition and suggest opportunities for improving search efficiency in LLM code generation.

• The paper presents a layer-wise analysis showing how expert overlap varies by token identity, challenging the context-independent routing hypothesis.
• It employs a novel compiler-output alignment methodology to isolate semantically meaningful code differences in over 851 C code samples.
• The study highlights that LLM code generation largely produces superficial formatting variations, underscoring the need for advanced decoding strategies.

Layer-wise Analysis of MoE Routing Locality in Shared-Prefix Code Generation

Summary and Core Contributions

This paper provides a systematic, quantitative study of Mixture-of-Experts (MoE) routing locality in LLM code generation under shared-prefix tree-search decoding. Using Qwen3.5-35B-A3B-FP8 with 256 experts and top-8 routing, the authors investigate the extent to which expert sets overlap across candidate generations branching from a common prompt, decomposing these effects by model layer and token identity. The primary methodological innovation is the use of compiler-output-based (gcc -S -O0) alignment to remove token-identity artifacts and focus on semantically meaningful code diversity. Key results challenge and refine the common “context-independent routing” assumption from prior MoE work, revealing a nuanced, layer-dependent pattern of routing overlap. The work also quantitatively demonstrates that top-$P$ and beam search diversity in LLM code generation is largely superficial, dominated by comments and blank lines.

Experimental Design and Methodology

The experiments are conducted in the context of C code generation using a prompt that encourages clean, comment-free code. A custom tree-search generation framework identifies decision points where multiple candidate tokens can be chosen, generating 851 completed code samples branching from a shared prefix. Compiler-based alignment assigns codes into equivalence groups where assembled output is identical, enabling precise measurement of both within-sample and between-sample routing overlap, independent of formatting variation.

Per-token, per-layer routed expert IDs are logged for all completions. Jaccard similarity is used to quantify the overlap in active expert sets, and these statistics are reported for positions aligned as same-token or different-token, both within and between O0-equivalent (assembly-identical) groups. Additional decomposition is performed by token type (keywords, identifiers, operators, etc.) and by specific model layers, including both GDN and full-attention layers. Tree branching dynamics, including decay of overlap with distance from the fork, are empirically characterized.

Principal Results

MoE Routing Locality and Layer-Wise Structure

• Expert overlap per token position, even at different-token sites, greatly exceeds the random baseline. For instance, the mean Jaccard similarity for different-token positions is 0.175 (11$\times$ random), and for same-token positions is 0.649 (40$\times$ random).
• A crossing pattern is observed layer-wise: Same-token routing similarity is highest in input layers (L0 ≈ 0.8), dips mid-network (L14–L20 ≈ 0.6), and slightly recovers in deep layers. Conversely, different-token similarity is lowest at input but peaks in the middle layers (L14 ≈ 0.22, 14$\times$ random), consistent both within assembly-equivalent code and between distinct outputs.
• There is no significant difference in routing behavior between GDN and full-attention layers; the position in the layer stack, not the attention mechanism, is dominant.
• Expert co-activation matrices reveal structured, non-random specialization—a substantial fraction of expert pairs are never co-selected, and a few experts are responsible for the majority of activations.

Code Diversity and Redundancy

• Among 851 generated codes, only 189 compiler-equivalent (O0 groups) are observed—analysis of the largest groups reveals that 99.6% of within-group differences are due to comments and blank lines. Actual code logic differences (e.g., in algorithm selection) are extremely rare.
• Attempts to suppress comments in prompting are ineffective due to model “reasoning leakage”: when interim reasoning is suppressed in chat templates, the model embeds its explanatory process in post-code comments, which then drive branching in dead code selection.
• 67% of compiled codes are concentrated in the top three assembly-equivalent groups, and execution-relevant diversity is minimal—LLM beam and top-$P$ search primarily explore trivial surface-level variation under the studied settings.
• Branching tree expansion in stochastic search is dominated by such superficial differences, rapidly exhausting computational resources without meaningful semantic exploration.

Token-Type and Fork Dynamics

• Routing overlap is highly token-type dependent: keywords exhibit the greatest overlap (same-tok Jaccard 0.685), while identifier positions have the greatest context-dependent similarity in different-token alignments (Jaccard 0.193).
• In sibling decoding branches, routing overlap decays with steps since fork but remains far above random expectation even after significant divergence, implying robust shared-context bias.

Theoretical and Practical Implications

Refinement of “Context-Independent Routing” Hypothesis

The study directly challenges the blanket context-independence hypothesis posed by OpenMoE and others, showing instead a well-defined layer-dependent structure: token identity dominates routing in early layers, but context dependence peaks in the middle layers. Single-layer analyses miss this critical nuance, suggesting prior generalizations are oversimplified. This has implications for MoE interpretation, subnetwork adaptation, and information leakage analyses [see OpenMoE, (Xue et al., 2024)].

MoE Inference Optimization and Sharing

High inter-request routing locality, especially in the middle layers, implies substantial opportunities for expert offloading optimization. Shared-prefix batches can exploit predictable activation patterns for improved expert placement and hardware utilization, going beyond existing approximate approaches [cf. FMOE, XShare, (Yu et al., 7 Feb 2025, 2602.07265)].

Search Efficiency and Code Generation

The results starkly highlight the limitations of naive stochastic search in LLM code generation—true code logic diversity is low under vanilla sampling and beam search, with most resources spent on formatting variations. Efficient beam allocation and diversity enhancement likely demand runtime AST or compiler-based equivalence checks to prune redundant candidates. This is essential for code synthesis, evaluation, and deployment workflows using LLMs.

Model Behavior and Prompting

Code generation with LLMs remains highly sensitive to prompting and output format injection. The persistent comment leakage, even with explicit instructions, demonstrates the limitations of current prompt engineering and suggests avenues for improved decoding control and template handling.

Future Directions

• Extension to open-ended text and other programming languages, assessing routing locality and semantic diversity patterns outside C code.
• Model-agnostic replication, particularly across newer or architecturally distinct MoE implementations.
• Integration of runtime equivalence checks (AST/canonicalization/compiling) in decoding pipelines to maximize generation utility per unit cost.
• Direct translation of routing locality statistics into system-level MoE inference speedup simulations and deployment in large-scale environments.

Conclusion

This work establishes that MoE routing overlap, under shared-prefix code generation, demonstrates strong persistent locality—far exceeding random expectation—for both same-token and different-token positions, governed by a clear layer-wise transition between token-dominated and context-dominated regimes. It quantitatively shows that current decoding practices in LLM code generation predominantly explore surface-level code variants, offering little meaningful algorithmic diversity. These findings both refine prevailing assumptions about MoE behavior and inform new strategies for search, inference optimization, and generative diversity management in large-scale LLM systems.