Modular Representation Compression: Adapting LLMs for Efficient and Effective Recommendations

Abstract: Recently, LLMs have advanced recommendation systems (RSs), and recent works have begun to explore how to integrate LLMs into industrial RSs. While most approaches deploy LLMs offline to generate and pre-cache augmented representations for RSs, high-dimensional representations from LLMs introduce substantial storage and computational costs. Thus, it is crucial to compress LLM representations effectively. However, we identify a counterintuitive phenomenon during representation compression: Mid-layer Representation Advantage (MRA), where representations from middle layers of LLMs outperform those from final layers in recommendation tasks. This degraded final layer renders existing compression methods, which typically compress on the final layer, suboptimal. We interpret this based on modularity theory that LLMs develop spontaneous internal functional modularity and force the final layer to specialize in the proxy training task. Thus, we propose \underline{M}odul\underline{a}r \underline{R}epresentation \underline{C}ompression (MARC) to explicitly control the modularity of LLMs. First, Modular Adjustment explicitly introduces compression and task adaptation modules, enabling the LLM to operate strictly as a representation-learning module. Next, to ground each module to its specific task, Modular Task Decoupling uses information constraints and different network structures to decouple tasks. Extensive experiments validate that MARC addresses MRA and produces efficient representations. Notably, MARC achieved a 2.82% eCPM lift in an online A/B test within a large-scale commercial search advertising scenario.

• The paper introduces MARC, a modular framework that applies HSIC constraints to compress high-dimensional LLM representations without losing rich semantic features.
• It reveals that mid-layer representations outperform final layers for recommendation tasks, highlighting a structural advantage in model modularity.
• Experimental results show that MARC achieves superior performance and a notable eCPM lift in industrial-scale tests compared to baseline methods.

Modular Representation Compression for Integrating LLMs in Recommendation Systems

Motivation and Empirical Foundations

LLMs have increasingly been integrated into industrial-scale recommender systems (RSs) to encode user/item representations with rich semantics. However, the high dimensionality of LLM representations causes substantial overhead in storage, training, and inference, especially when scaled to millions of users and items. The empirical results display a strong correlation between larger LLMs and improved downstream performance, but this benefit is offset by prohibitive computational costs. Conventional dimensionality reduction strategies (e.g., PCA) are inadequate due to significant performance declines.

Figure 1: Downstream performance and cost implications as representation dimensions scale with different LLM backbones.

A pivotal observation documented in this work is the Mid-layer Representation Advantage (MRA): representations extracted from middle layers of LLMs consistently outperform those from final layers for recommendation tasks. This phenomenon persists across various fine-tuning objectives, including contrastive and CTR-based losses, indicating its structural origin rather than an objective alignment artifact.

Figure 2: MRA phenomenon showing superior performance from mid-layer representations across datasets and LLMs.

Modular Theory Interpretation and Methodological Innovations

The authors interpret MRA through the theoretical lens of modularity. During fine-tuning with recommendation-aligned objectives, LLMs spontaneously develop functional modularity—early and middle layers act as representation learners, while final layers shift to task adaptation, specializing for the explicit proxy training objective and filtering out richer semantics necessary for recommendation.

Layer-wise proxy task performance further evidences this modular specialization. While final layers achieve minimal loss regarding the proxy objective, their representations are less suited for multifaceted recommendation scenarios that require generalized information.

Figure 3: Final-layer representations exhibit optimality for proxy objectives but are suboptimal for downstream recommendations.

Existing compression techniques (nested- and projection-based) typically operate on final-layer outputs, thus yielding compressed embeddings from degraded representations. Moreover, selecting optimal layers for compression is untenable in real-world settings due to variability across tasks, models, and datasets.

MARC: Modular Representation Compression Framework

To counteract spontaneous, uncontrolled modularity, the Modular Representation Compression (MARC) framework is proposed. MARC explicitly enforces modular boundaries by introducing dedicated external modules: a compression network for representation compaction and a user-item matching network for task adaptation. Crucially, MARC applies task-specific losses only on the external modules, not directly on LLM outputs, thus preserving the backbone's generalization capabilities.

MARC leverages the Hilbert-Schmidt Independence Criterion (HSIC) as an information constraint for compression, maximizing mutual information between original and compressed representations without requiring dimensional alignment. The matching network performs explicit and implicit user-item interaction modeling, absorbing task adaptation pressure and isolating it from the LLM backbone.

Figure 4: Architectural contrast between conventional projection-based compression (left) and MARC with externally decoupled modules (right).

Experimental Validation

MARC achieves superior empirical results against both training-free and fine-tuning compression baselines on CTR, re-ranking, and retrieval tasks. The compressed 128-dimensional representations frequently perform on par with or better than uncompressed, high-dimensional outputs from frozen LLMs. Notably, in an industrial-scale production A/B test, MARC delivered a significant 2.82% lift in eCPM over a strong baseline in a commercial search advertising scenario.

Ablation studies demonstrate that removal of any MARC module—especially the HSIC constraint—significantly deteriorates performance, confirming the effectiveness of its modular architecture. The modular design, rather than loss function design or network interaction specifics, is the primary factor alleviating MRA and enabling robust, task-agnostic representations.

Figure 5: Effect of modular components and HSIC constraint in MARC via ablation study.

Figure 6: MARC consistently alleviates MRA across loss functions, confirming structural advantage.

Compatibility analysis verifies MARC's versatility across backbone LLMs and downstream models, with consistent mitigation of MRA and retention of performance, regardless of dimensionality.

Figure 7: MARC-encoded representation quality across multiple backbone LLMs, including comparisons to frozen LLM outputs.

Figure 8: MARC sustains superior performance across a range of compressed dimensions, outperforming baselines consistently.

Implications and Future Directions

The MARC framework addresses a fundamental challenge in LLM-integrated recommendation systems by decoupling representation learning from task adaptation, thus enabling efficient, high-density, low-dimensional embeddings suitable for industrial deployment. Practically, MARC’s approach facilitates scaling LLM-based RSs to millions of users/items while respecting latency constraints—critical for real-time online serving.

Theoretically, MARC provides evidence for emergent modularity in deep networks and demonstrates the importance of explicit modular design for task transfer and generalization. With HSIC-based information constraints, MARC opens new directions for non-linear mutual information preservation in cross-task representation compression.

MARC also suggests broader applicability: modular compression may benefit other domains requiring efficient LLM integration (e.g., retrieval, personalization, multi-task learning). Future developments could further optimize compression-network architectures, explore more advanced independence criteria, and extend modular boundaries to dynamic task-switching scenarios in RSs and beyond.

Conclusion

This paper establishes modular representation compression as a critical paradigm for scaling LLM-enhanced recommender systems. By exposing and addressing the mid-layer representation advantage via explicit modularity control, MARC demonstrates superior efficiency and effectiveness, validated by both strong offline numerical gains and substantial real-world impact. The modular decoupling strategy and HSIC-driven compression signal important progress for deploying LLM-backed RSs under practical constraints and enables richer, generalized embeddings for a wide array of downstream tasks.