Model size threshold where compressed representations surpass raw text

Determine whether there exists a threshold in large language model parameter count beyond which the compressed continuous document representations (memory-token embeddings) used for retrieval-augmented generation outperform using raw text context in supporting model understanding and answer generation.

Background

CLaRa compresses documents into continuous memory-token representations and trains retrieval and generation jointly within a shared latent space. Experiments in this work use medium-scale backbones (Mistral-7B and Phi-4B) and show that high-quality compression can match or even surpass text-based baselines in some settings.

The authors explicitly raise a scaling question: as model size grows, it is unclear whether and when compressed representations could consistently outperform raw text for understanding and generation. Establishing such a threshold would guide the choice between compressed latent inputs and full-text prompts for future retrieval-augmented systems.