Code Generation for Compression

• Compression by code generation is a technique that synthesizes and refactors code to maximize information density while preserving or enhancing target task performance.
• It employs methods like prompt compression, model-level quantization, and symbolic library induction to reduce computational overhead and improve efficiency.
• Benchmarks such as the KoLMogorov Test and strategies like LoRD and CodePromptZip illustrate both significant advances and current challenges in capturing optimal algorithmic compressibility.

Compression by code generation denotes a class of approaches where code—either as program snippets, prompts, model architectures, or intermediate representations—is systematically generated, refactored, or learned to maximize information density while minimizing resource consumption, all with the overarching goal of preserving or enhancing target task performance. This paradigm spans applications in program synthesis, prompt optimization, retrieval-augmented generation (RAG), model compression, and theoretical benchmarks inspired by Kolmogorov complexity, each leveraging code as an explicit or implicit vehicle for information compression, abstraction, and efficient computation.

1. Formal Foundations and Theoretical Perspective

The core theoretical ideal motivating "compression by code generation" is the minimization of descriptive or algorithmic complexity, exemplified by the Kolmogorov complexity of a string $x$: $K(x)=\min_{p:U(p)=x}|p|$, the length of the shortest program $p$ that outputs $x$ on a reference universal Turing machine $U$ (Yoran et al., 18 Mar 2025). Achieving $K(x)$ is uncomputable, but this metric provides a gold standard for data compression as program synthesis. In practical settings, compression by code generation is operationalized by synthesizing short, correct, and readable code artifacts that, when executed, reproduce target sequences, behaviors, or abstractions, implicitly upper-bounding $K(x)$.

Benchmarks such as the KoLMogorov Test (KT) instantiate this ideal, evaluating code-generating LLMs by their ability to produce the shortest correct code for input data sequences (Yoran et al., 18 Mar 2025). This paradigm is universally applicable (audio, text, DNA), with correctness and compressed code length as uncompromisable evaluation metrics. However, state-of-the-art LLMs substantially underperform classical compressors (e.g., Gzip) on real data, highlighting the challenge of learning compositional or algorithmic patterns from naturalistic input.

2. Prompt, Instruction, and Context Compression

In code generation, input compression can be realized at the prompt, documentation, or context level, directly influencing computational efficiency and cost.

DocString Compression: ShortenDoc exemplifies task-specific prompt compression by removing non-informative DocString tokens from code-generation prompts while enforcing a similarity constraint on the generated code distribution's pre-softmax logits (cosine similarity $\tau\approx0.999$) (Yang et al., 30 Oct 2024). Tokens are ranked via their impact on code-LM perplexity, with low-importance n-grams greedily removed if the quality constraint is satisfied. ShortenDoc achieves 25–40% compression with no reduction—and sometimes improvement—in pass@1 solution quality, and ∼17–20% decrease in FLOPs.

Code-based Prompt Compression in RAG: CodePromptZip addresses the context window and resource bottleneck of retrieval-augmented code generation. Input code examples are program-analytically decomposed and tokens are ranked type-wise (Symbol, Signature, Invocation, Identifier, Structure) by ablation-driven utility. A CodeT5-based compressor, conditioned on desired compression ratios and augmented with a pointer-generator mechanism, outputs compressed code that maintains task performance; empirical results indicate up to 30% code reduction with 8–28% relative downstream performance gains over state-of-the-art entropy or distillation-based methods (He et al., 19 Feb 2025).

Continuous Code Representation Compression: LlavaCode projects retrieved code snippets into condensed embedding vectors via a lightweight projector MLP, reducing thousands of tokens per example to a single embedding. This context, prepended as “soft tokens” to the main prompt, enables 20–38% reduction in time-to-first-token with negligible loss in EM/ES accuracy for line completion (Cherniuk et al., 22 Oct 2025).

3. Token and Sequence Compression in Multimodal and Sequential Pipelines

For vision–language and UI-to-code LLMs, high-efficiency sequence compression is required on both the input and output sides due to the vast number of visual tokens and the verbosity of generated code.

EfficientUICoder introduces a three-stage compression pipeline (Xiao et al., 15 Sep 2025):

• Element- and Layout-aware Token Compression (ELTC): Detects UI element trees and prunes visual tokens outside element regions via minimum spanning tree construction.
• Region-aware Token Refinement (RTR): Further prunes or supplements tokens within element regions by their attention-based salience.
• Adaptive Duplicate Token Suppression (ADTS): Controls repetition in generated HTML/CSS using on-the-fly frequency tracking and exponential penalty to logits.

Integrated, this approach yields a 55–60% visual-token compression ratio, 41.4% fewer generated tokens, and nearly 50% speedup in prefill/inference time on 34B-parameter MLLMs while preserving HTML fidelity.

4. Model-level Compression: Weights and Attention Mechanisms

Compression by code generation also manifests in model-level parameter reduction, targeting the storage and compute profiles of code LLMs.

Low-Rank Decomposition (LoRD): For large code-generation models (e.g., StarCoder), LoRD factors each linear layer’s weight matrix $W$ into $$U \in \mathbb{R}^{d_1 \times r}, V \in \mathbb{R}^{r \times d_2}$$ with $r\ll\min(d_1,d_2)$, initialized by approximate SVD or eigen-analysis (Kaushal et al., 2023). No retraining is needed. Up to 39.6% parameter reduction (<1% perplexity increase), 17.5% GPU RAM reduction, and 22.35% inference speedup are empirically demonstrated; LoRD’s decomposed models remain fully differentiable and are compatible with state-of-the-art quantization (SpQR) and parameter-efficient fine-tuning (QLoRA).

Post-Training Quantization: Quantizing models to int8 per-column or per-tensor (activation and weight matrices) after static or dynamic calibration yields ∼70% model size reduction and halves CPU inference latency even for 6B–16B parameter models, all with <2 percentage point decrease in pass@1 accuracy and no meaningful drop in robustness (Wei et al., 2023).

Attention State Compression: AnchorCoder analyzes attention sparsity in code LLMs and compresses key-value caches by introducing anchor tokens at semantic boundaries (e.g., line breaks), storing only anchor KV-states per layer, and propagating critical context via cross-layer anchor attention (Zhang et al., 11 Nov 2024). This strategy cuts KV cache memory by at least 70%, decouples per-layer complexity from input length, and preserves 96–102% of baseline pass@1 performance at typical compression levels.

5. Program Synthesis, Library Learning, and Symbolic Compression

Code generation itself can be the substrate of compression, via explicit synthesis, refactoring, and library abstraction.

Neuorsymbolic Library Induction: LILO alternates neural program synthesis and symbolic compression (Stitch), greedily mining repeated subprograms from a corpus, scoring $$\Delta(f) = \sum_{p}|p|-|rewrite(p;\mathcal{B}_0\cup\{f\})| - |f|$$, and naming abstractions via LLM-powered auto-documentation. This minimizes total description length and yields compact, interpretable libraries. Empirically, LILO achieves 1.5–3.5× compression of program corpora and higher solve rates on synthesis benchmarks relative to both neural-only and symbolic-only baselines (Grand et al., 2023).

Data Compression via Program Synthesis: The KoLMogorov Test (Yoran et al., 18 Mar 2025) frames compression as synthesis of the shortest correct program for a data sequence; current code LLMs underperform, often resorting to memorization rather than semantic abstraction, indicating that code-based compression remains bounded by model reasoning capacity and algorithmic generalization.

6. Specialized Code Compression in Embedded and Statistical Settings

Access-Pattern Code Compression for Embedded Systems: Code can be compressed at the granularity of basic blocks (using CFGs), with online decompression and recompression triggered by predicted execution traces (e.g., k-edge lookahead). Although the approach is heuristic and does not deliver hard real-time bounds or comprehensive performance metrics, it provides a template for code-space reduction integrated with application behavior (0710.4799).

Generator Matrix as Compressor: In statistical settings, the generator matrix $G$ of a binary linear code can act as a compression mapping from $n$-bit random vectors to $k$-bit outputs, with the output distributions (Walsh-Hadamard transforms) statistically revealing weight distributions of the code. This is a lossy compressor in the information-theoretic sense, but the sampling approach is tractable only for codes of modest length (Tomasi et al., 2018).

7. Limitations, Open Problems, and Synthesis

Despite significant advances, compression by code generation faces fundamental limitations:

• Generalization from synthetic to natural data remains inadequate for existing program-synthesis LLMs; the underlying abstraction and reasoning demands exceed current model capacity (Yoran et al., 18 Mar 2025).
• Model-level compression methods (LoRD, quantization, anchor attention) optimize for resource efficiency but do not mitigate algorithmic limitations around reasoning depth or compositionality, and may require task-specific calibration (Wei et al., 2023, Kaushal et al., 2023, Zhang et al., 11 Nov 2024).
• Prompt and code-context compressors rely on high-quality token importance measures and typically require language- or task-specific tuning for maximal efficacy (Yang et al., 30 Oct 2024, He et al., 19 Feb 2025).
• Symbolic abstraction strategies (library/DSL induction) demonstrate compression wins in controlled domains but require significant orchestration between search, abstraction mining, and nomenclature mechanisms for interpretable downstream deployment (Grand et al., 2023).

A plausible implication is that hybrid strategies—blending explicit search/planning, neural code generation, symbolic refactoring, and fine-grained data/context compression—will be required to systematically close the empirical gap with algorithmic-optimal (Kolmogorov) compressibility benchmarks. Continued progress on joint learning and search, robust information-aware compression modules, and execution- or reward-driven training loops is likely to define the next generation of compression by code generation methods.