Cross-Layer Expert Sharing in Complex Systems

Updated 25 July 2025

Cross-layer expert sharing is a methodology that integrates specialized submodules across different system layers to foster coordinated decision-making and resource reuse.
It employs techniques such as dynamic parameter sharing, basis sharing via SVD, and KV cache sharing to enhance throughput and reduce memory usage.
Key trade-offs include balancing efficiency gains with potential accuracy degradation, which are mitigated through compensation mechanisms and careful layer selection.

Cross-layer expert sharing refers to methodologies for enabling the exchange, integration, or reuse of specialized computational submodules (“experts”) or critical information across different abstraction layers in complex systems. Originally emerging from networked communications, the principle now spans domains such as distributed wireless networks, vision/LLM adaptation, and large-scale neural network inference. At its core, cross-layer expert sharing aims to enhance system efficiency, performance, and adaptability by overcoming traditional strict separation between layers—facilitating coordinated decision-making, parameter/resource sharing, and heterogeneous module collaboration.

Cross-layer expert sharing can manifest as:

Sharing of specific computational outputs or parameters (e.g., attention weights, key-value caches, SVD-derived bases) across neural network layers to reduce redundancy and improve efficiency (Mu et al., 4 Aug 2024, Brandon et al., 21 May 2024, Wang et al., 2 Oct 2024, Wu et al., 18 Oct 2024, Yang et al., 24 Oct 2024, Wen et al., 10 Jul 2025).
Cross-layer information propagation in protocols or network stacks, where insights from lower layers (physical or MAC) guide upper-layer strategies (e.g., resource allocation, routing, or application-layer adaptation) (0704.2841, 1206.5459, Baligh et al., 2014, Jagannath et al., 2021, Liu et al., 7 Jun 2024, Zhang et al., 2016).
Expert module integration in Mixture-of-Experts (MoE) or federated settings, where models or modules trained at one layer/domain are reused or aggregated across domains/layers for improved adaptability or privacy (Yao et al., 16 Jan 2024, Liu et al., 17 Mar 2025, Qin et al., 17 Mar 2025, Li et al., 10 Apr 2025).

A unifying feature is the move from isolated, independently optimized modules towards sharing of “expert” structures or information either horizontally (within a layer, across distributed agents) or vertically (across layers of abstraction).

2. Architectures, Algorithms, and Methodologies

A variety of architectures operationalize cross-layer expert sharing, including:

Expert Affinity and Placement: In distributed MoE inference, “inter-layer expert affinity”—the conditional probability that tokens routed to a given expert in one layer remain with related experts in the next—enables clustering of highly affiliated experts on the same device or node. Optimization reduces inter-device communication, leveraging integer programming to align expert placement with affinity (Yao et al., 16 Jan 2024).
Dynamic Parameter/Activation Sharing: Methods such as Cross-Layer Attention (CLA) and LiSA share key/value projections or attention weights across layers instead of computing them anew in each layer. CLA does so in a fixed pattern (e.g., every two layers share KV caches) (Brandon et al., 21 May 2024), while LiSA uses a learnable alignment module and low-rank corrections to replace direct per-layer computation, maintaining accuracy with fewer parameters (Mu et al., 4 Aug 2024).
Basis Sharing via SVD: Weight matrices from multiple layers are approximated as linear combinations of a small set of shared basis vectors plus per-layer coefficients, discovered using SVD. This “basis sharing” achieves high compression with minimal accuracy loss, especially when sharing is limited to carefully selected layer types or adjacent layer pairs (Wang et al., 2 Oct 2024).
KV Cache Sharing: Unified frameworks, such as the one in (Wu et al., 18 Oct 2024), generalize approaches where non-KV layers consume key/value pairs from specific “KV layers,” supporting a range of sharing configurations (bottom-, top-, or middle-layer sharing), depending on whether the queries are paired with KVs from lower, upper, or central layers.
Dynamic, Context-Adaptive Sharing: Krul dynamically selects which layers to compress by analyzing inter-layer attention similarity, favoring sharing in layers where this is unlikely to harm future context retention, thus customizing the sharing strategy to conversation-specific dynamics (Wen et al., 10 Jul 2025). A token-wise heterogeneous estimator and a coordinated restoration scheduler maintain both efficiency and quality during conversation resumption.
MoE Pathway Optimization: C3PO reframes expert pathway selection as a test-time optimization, adjusting routing weights layer-wise for each test instance using surrogate objectives derived from “successful neighbor” samples, focusing computation on critical layers and core experts, thus enabling collaborative improvement beyond static pretraining pathways (Li et al., 10 Apr 2025).
Federated Cross-Domain MoE: In recommendation settings with strict privacy or non-overlapping users, sharing is restricted to model checkpoints/parameters between disjoint domains (clients), enabling knowledge transfer across heterogeneous domains by aggregating local “experts” via federated learning and mixtures-of-experts gating, but never raw user data (Liu et al., 17 Mar 2025).

3. Efficiency, Throughput, and Performance Trade-offs

Cross-layer expert sharing typically yields:

Reduced Memory and Computation: By reusing or compressing activations, parameters, or KV caches across layers, memory footprints are lowered (e.g., 2× reduction in KV cache with CLA (Brandon et al., 21 May 2024), 30% cache cut with KVSharer (Yang et al., 24 Oct 2024), 6× QK compression with LiSA (Mu et al., 4 Aug 2024)). In MoE inference, alignment of expert affinity saves up to 67% of cross-GPU routing latency (Yao et al., 16 Jan 2024).
Throughput Improvement: Experimentally, throughput gains include up to 32% faster attention computation for LiSA (Mu et al., 4 Aug 2024), 2.2× higher MoE inference with ExFlow (Yao et al., 16 Jan 2024), 1.5–2.68× lower TTFT for Krul (Wen et al., 10 Jul 2025), and 1.3–1.6× speed-up in token generation with KVSharer (Yang et al., 24 Oct 2024).
Minimal Degradation or Even Gains in Accuracy: Most methods report negligible drops in perplexity or downstream accuracy when sharing ratios are moderately aggressive, particularly with careful selection or learnable compensation (e.g., LiSA’s low-rank correction in sensitive layers). However, over-aggressive sharing, especially without compensation or proper partitioning, may lead to loss of specialization and reduced downstream performance (Wu et al., 18 Oct 2024).
Optimized Resource Allocation: In wireless and networking contexts, sharing enables more balanced trade-offs across throughput, energy efficiency, and latency by integrating MAC/PHY data into higher-layer adaptation and resource decisions (0704.2841, Baligh et al., 2014, Jagannath et al., 2021, Liu et al., 7 Jun 2024).

4. Design Considerations, Limitations, and Challenges

Critical implementation factors include:

Careful Sharing Granularity: Over-broad sharing (e.g., tying too many layers to a single basis in basis sharing (Wang et al., 2 Oct 2024), or merging high-similarity KV caches (Yang et al., 24 Oct 2024)) may harm expressivity and degrade performance. Selecting which layers, parameter types, or groups are eligible for sharing is typically driven by measures such as Frobenius loss, accuracy drop, or empirical attention similarity.
Compensation Mechanisms: Lightweight correction modules (e.g., low-rank projections in LiSA, local coefficients in basis sharing, or compensation terms in shared experts) are frequently required to recover per-layer specialization.
Dependency on Training or Calibration Data: Some methods require access to calibration data (e.g., for computing cache similarity matrices as in KVSharer (Yang et al., 24 Oct 2024) or Krul (Wen et al., 10 Jul 2025)) for optimal sharing decisions, which may impact adaptability in resource-constrained or privacy-limited environments.
Pipeline and Hardware Implications: Cross-layer sharing can affect parallelism and scheduling. For instance, non-bottom sharing configurations in KV sharing may require iterative training and inference, leading to additional prefilling latency (Wu et al., 18 Oct 2024). Efficient hardware support for sharing, dynamic loading, and recomputation (as orchestrated in Krul (Wen et al., 10 Jul 2025)) is essential for maximizing efficiency.

5. Applications in Communications, Networks, and Distributed Inference

Cross-layer expert sharing is applied in:

Distributed Wireless Networks: Combining physical-layer beamforming with MAC-level coordination through shared collision resolution and channel estimation accelerates communication and enhances energy efficiency (0704.2841, Jagannath et al., 2021). Joint cross-layer frameworks for 5G likewise optimize admission, association, power control, scheduling, and routing using shared optimization algorithms (e.g., WMMSE) (Baligh et al., 2014).
Adaptive Multimedia Delivery: StreamOptix balances PHY, MAC, and application-layer constraints for video streaming quality by sharing measurements (such as BLER and channel capacity) upward to enhance bitrate adaptation and resource allocation (Liu et al., 7 Jun 2024).
Cyber-Physical and Real-Time Systems: Cross-layer profiling tools such as X-Lap integrate timing measurements at application, OS, and protocol layers to identify and mitigate sources of jitter or delay (Reif et al., 2018).
Federated Learning and Privacy-Preserving Recommendation: Only parameter checkpoints or expert models are shared across domains/clients, enabling collaborative expert aggregation without transferring sensitive user data (Liu et al., 17 Mar 2025).
Edge AI on Wireless Networks: Expert and resource allocation decisions are jointly managed across model and physical layers to balance inference accuracy and communication energy, using formal optimization and selection algorithms (Qin et al., 17 Mar 2025).

6. Future Directions and Open Problems

Several research avenues are open:

Dynamic and Contextual Sharing Schemes: Methods such as Krul (Wen et al., 10 Jul 2025) that tailor cross-layer sharing strategies dynamically to input distributions or conversation histories signal a move away from static compression toward context-aware, adaptive schemes.
Generalization Across Modalities and Architectures: The principles of cross-layer sharing demonstrated in vision/LLMs, network protocols, and MoE architectures suggest applicability to broader domains (e.g., multi-modal transformers, distributed sensor inference).
Composable and Hybrid Compression Techniques: Combining cross-layer sharing with intra-layer compression, quantization, pruning, and other parameter efficiency methods may deliver further improvements, as explored in (Yang et al., 24 Oct 2024).
Balance of Specialization and Efficiency: Determining the optimal sharing granularity and compensation for preserving task specialization versus achieving maximum efficiency remains a fundamental challenge, especially as models grow deeper and more heterogeneous.

7. Summary Table: Major Techniques and Their Features

Method/Paper	Domain	Shared Elements	Sharing Strategy	Key Benefit
CLA (Brandon et al., 21 May 2024), LCKV (Wu et al., 18 Oct 2024)	LLM inference	KV cache	Adjacent/fixed pattern	2×+ memory reduction, high throughput
KVSharer (Yang et al., 24 Oct 2024)	LLM inference	Layer KV caches	Layer-wise, dissimilar-based	30%+ cache cut, 1.3×–1.6× speedup
LiSA (Mu et al., 4 Aug 2024)	LLM inference	Attention weights	Learnable, low-rank compensation	6× Q/K reduction, up to 32% faster
Basis Sharing (Wang et al., 2 Oct 2024)	LLM compression	Weight matrices	SVD basis + per-layer coefficients	High compression, minimal PPL loss
ExFlow (Yao et al., 16 Jan 2024)	Distributed MoE	Routing/experts placement	Affinity-based token routing	67% comm. reduction, up to 2.2× throughput
C3PO (Li et al., 10 Apr 2025)	MoE LLM adaptation	Expert routing weights	Test-time, critical layers + core experts	7–15% accuracy improvement
FMoE-CDSR (Liu et al., 17 Mar 2025)	Federated recommender systems	MoE checkpoints/models	Privacy-preserving, parameter-only	Domain transfer w/o user overlap
Dynamic DMoE (Qin et al., 17 Mar 2025)	Distributed edge AI	Experts, subcarriers	Joint selection, energy/task tradeoff	High accuracy, efficient resource use
StreamOptix (Liu et al., 7 Jun 2024)	Video streaming over wireless	PHY/MAC metrics to APP	Closed-loop resource adaptation	Better QoE, BER/PSNR optimization