An Information Theoretic Perspective on Agentic System Design (2512.21720v1)

Abstract: Agentic LLM (LM) systems power modern applications like "Deep Research" and "Claude Code," and leverage multi-LM architectures to overcome context limitations. Beneath their apparent diversity lies a recurring pattern: smaller "compressor" LMs (that can even run locally) distill raw context into compact text that is then consumed by larger "predictor" LMs. Despite their popularity, the design of compressor-predictor systems remains largely ad hoc, with little guidance on how compressor and predictor choices shape downstream performance. In practice, attributing gains to compression versus prediction requires costly, task-specific pairwise sweeps. We argue that these agentic system design questions are, at root, information-theoretic. Viewing the compressor LM as a noisy channel, we introduce a simple estimator of mutual information between the context and its compression to quantify compression quality in a task-independent way. We show that mutual information strongly predicts downstream performance, independent of any specific task. Through an information-theoretic framework, we perform a comprehensive empirical analysis across five datasets and three model families. Results reveal that larger compressors not only are more accurate, but also more token-efficient, conveying more bits of information per token. A 7B Qwen-2.5 compressor, for instance, is $1.6\times$ more accurate, $4.6\times$ more concise, and conveys $5.5\times$ more bits of mutual information per token than its 1.5B sibling. Across datasets, scaling compressors is substantially more effective than scaling predictors, enabling larger on-device compressors to pair with smaller cloud predictors. Applied to a Deep Research system, these principles enable local compressors as small as 3B parameters to recover $99\%$ of frontier-LM accuracy at $26\%$ of API costs.

• The paper demonstrates that scaling compressor models significantly enhances downstream QA accuracy and cost efficiency.
• It frames the agentic LM pipeline as a noisy channel, using mutual information as a key measure of compression fidelity.
• Empirical results confirm that higher compressor quality, reflected in better token-efficiency and MI, drives overall system performance.

An Information-Theoretic Framework for Agentic System Design

Introduction

This work presents an information-theoretic approach to the architectural design of agentic LLM (LM) systems, synthesizing empirical and theoretical insights to formalize the trade-offs in compressor-predictor pipelines. The paper operationalizes mutual information (MI) as a predictor of downstream utility, decoupling the evaluation of compression and reasoning stages in multi-LM agentic workflows commonly seen in applications such as Deep Research and automated reasoning pipelines. The study establishes the primacy of compressor quality—specifically, the capacity and token-efficiency of the compressor LM—as the main driver of end-to-end system accuracy and cost efficiency, and validates these principles across a broad suite of tasks, datasets, and model families.

Figure 1: Many agentic LM systems use compressors to distill long contexts into compact summaries $Z$ consumed by predictors, with consumer hardware increasingly supporting larger open-weight LMs for on-device compression.

Theoretical Setup: Compression as a Noisy Channel

The foundational insight is to reframe the agentic system pipeline—where a smaller compressor LM summarizes input context $X$ into $Z$, then a larger predictor LM processes $Z$ to yield answer $Y$—as an information bottleneck under the lens of classical rate-distortion theory:

$X \rightarrow Z \rightarrow Y$

The compressor $p(z|x)$ is modeled as a noisy communication channel, with the mutual information $I(X; Z \mid Q)$ (conditioned on query $Q$) serving as a model- and task-agnostic surrogate for quantifying compression fidelity. The authors introduce a practically computable MI estimator targeting autoregressive LMs, eliminating the need for full vocabulary log-probabilities and enabling large-scale empirical study. Rate (bit-efficiency) is defined by $R = I(X;Z \mid Q)/L$, $L$ being the output token length.

Empirical Study: Scaling Laws and Information Flow

To ground these concepts, the study executes a comprehensive scaling analysis across five datasets (LongHealth, FinanceBench, QASPER, WildChat, FineWeb) and three model families (Qwen-2.5, Llama-3, Gemma-3). Evaluation probes four principal questions: (1) whether to invest compute in compressor or predictor, (2) how compressor scale affects token- and compute-efficiency, (3) how MI and rate-distortion statistics correlate with performance, and (4) the relative importance of system design knobs.

Downstream Performance as a Function of Compression

Scaling the compressor LM yields markedly greater gains for downstream QA accuracy than scaling the predictor LM. On LongHealth, increasing Qwen-2.5 from 1B to 7B parameters improves accuracy by 60%, while scaling the predictor from 70B to 405B yields only a 12% boost.

Figure 2: Downstream accuracy, compression output length, and compute cost (GFLOPs-per-compression) all improve with compressor model size; larger compressors are simultaneously more accurate and more concise, especially evident on LongHealth and FinanceBench.

Furthermore, larger compressors are more token-efficient—outputting up to $4.6\times$ more concise summaries—imparting sublinear scaling in FLOPs-per-generation versus parameter count. This effect holds across multiple datasets and model families.

Figure 3: Scaling compressors delivers larger accuracy gains per compute (FLOPs-per-generation) than scaling predictors; predictor scaling saturates quickly.

Information Retention and Bit Efficiency

MI between context and compression, $I(X;Z|Q)$, increases as compressor scale increases, and so does bit efficiency (MI per output token). On LongHealth and FinanceBench, the largest compressors saturate the MI estimator’s theoretical maximum; scaling effects are robust to prompt ablations and task heterogeneity.

Figure 4: Larger compressors generate outputs with higher mutual information and bit efficiency, even as compression length drops; MI approaches the theoretical maximum at sufficient scale.

Prompt ablation demonstrates that accuracy and MI are largely invariant to explicit conciseness constraints; enforced shorter outputs yield higher efficiency without loss of retained information.

Figure 5: Scaling trends in accuracy, compute cost, and MI are consistent across conciseness instructions—efficiency gains stem from model capacity, not prompt engineering.

Rate-Distortion and Task-Agnostic Performance Prediction

Rate-distortion analysis shows that bit-efficiency is a strong predictor of QA accuracy and output perplexity across compressor-predictor configurations and datasets. The rate-distortion curve follows an exponential decay, confirming the information bottleneck formalism’s utility for system tuning without exhaustive task-specific sweeps.

Figure 6: Left—bit efficiency (rate) is highly predictive of distortion (error); Right—mutual information linearly correlates with perplexity (r = -0.84, $R^2 = 0.71$), demonstrating its value as a task-agnostic quality signal.

Crucially, there is no consistent synergy between compressor and predictor model families; cross-family pairings do not underperform, and the bottleneck is dominated by compressor quality.

Feature Attribution and Meta-Analysis

A generalized linear model analysis attributes the variability in system accuracy primarily to compressor model family and size, with predictor capacity contributing marginally after a basic threshold. Qwen-2.5 compressors are more compute-efficient and effective than their Llama and Gemma-3 counterparts.

Practical Implications: Deep Research At Scale

Applying the developed framework to a Deep Research pipeline, the authors demonstrate that a local 3B parameter compressor recovers 99% of frontier-LM accuracy at only 26% of the API cost—highlighting a major shift towards feasible on-device agentic computation.

Figure 7: (Left) RACE score versus cost shows that larger compressors can match or exceed frontier-LM baselines with dramatically reduced API expenditures. (Right) RACE scores improve with compressor scale across Llama predictor sizes.

Limitations and Directions for Future Work

The study acknowledges estimation limitations for MI at the smallest model scales (1–3B), the focus on GPT-style non-reasoning LMs, and the need for further analysis of iterative and reasoning-augmented workflows. Future research ought to refine MI estimation (e.g., InfoNCE bounds), explore rate-distortion-informed training objectives, extend the framework to function-calling and more structured communication formats, and analyze the impact of deployment-specific optimization.

Conclusion

This study provides a rigorous information-theoretic methodology for agentic system design, demystifying the attribution of accuracy and cost within compressor-predictor modularity. Empirical findings unambiguously establish that:

• Scaling compressors is more effective and more compute-efficient than scaling predictors, with larger compressors producing both more accurate and more concise outputs.
• Mutual information and bit efficiency are practical, task-agnostic proxies for downstream performance and should guide system tuning.
• Compressor model family and scale dominate the contribution to system performance, superseding the impact of predictor advances once a baseline capacity is reached.

These principles present a clear prescription: allocate compute to local compressors, optimize for information density, and select model families that maximize communication efficiency. This information-theoretic perspective stands to inform the next generation of on-device agentic architectures and modular LM workflows.

(2512.21720)