Semantic Density Effect (SDE): Maximizing Information Per Token Improves LLM Accuracy

Abstract: We introduce the Semantic Density Effect (SDE): the empirical finding that prompts carrying higher semantic information per token consistently produce more accurate, focused, and less hallucinated outputs across all major LLM families. SDE is defined as the ratio of semantically loaded tokens to total prompt tokens, adjusted for redundancy and concreteness. Unlike prior prompt optimization techniques that add tokens (Chain of Thought), duplicate the prompt (Prompt Repetition), or reorder components (Instruction Placement Effect), SDE improves performance by removing or replacing low-information tokens while preserving or sharpening the semantic signal. Evaluated across five frontier models and seven benchmarks, ultra-dense prompts (SDE > 0.80) outperform diluted counterparts by an average of +8.4 percentage points with 0 additional tokens and 0 latency overhead. Combined with Instruction Placement Effect (IPE), the gain reaches +11.7 percentage points

• The paper introduces SDE as a novel framework that maximizes information per token, leading to an average accuracy improvement of 8.4 percentage points.
• The methodology employs controlled experiments across five state-of-the-art LLMs, revealing that ultra-dense prompts, which are 34% shorter, yield additive gains with IPE.
• Ablation studies confirm that enhancing semantic density improves computational efficiency and model robustness, offering actionable heuristics for prompt engineering.

Semantic Density Effect: Information-Maximization in Prompt Engineering for LLMs

Introduction

The paper "Semantic Density Effect (SDE): Maximizing Information Per Token Improves LLM Accuracy" (2604.17659) provides a formal and empirical framework for prompt optimization in LLMs by systematically analyzing and quantifying the impact of semantic density on model performance. The authors introduce SDE as an independent and additive variable in prompt design, demonstrate its superiority over standard approaches, and provide grounded operational heuristics and ablation analyses.

Semantic Density Effect (SDE) Definition and Theoretical Motivation

SDE is formulated as the normalized ratio of information-bearing, non-redundant, concrete tokens to the total context length, adjusted for redundant and abstract content. The Semantic Density Score is defined as:

$$SDE(P) = \frac{S(P)}{W(P)} \times (1-R(P)) \times C(P)$$

where $S(P)$ is the count of unique, task-relevant semantic tokens, $W(P)$ is the prompt length, $R(P)$ is the redundancy penalty, and $C(P)$ the concreteness fraction. The theoretical basis for SDE is rooted in transformer attention dynamics: tokens compete for a fixed attention distribution, and the introduction of semantically dilute tokens (fillers, hedges, redundant restatements) results in an effective attenuation of actionable signal by diluting attention. This is orthogonal to structural prompt optimizations such as Instruction Placement Effect (IPE), which reorders rather than prunes or sharpens semantic content.

Experimental Methodology and Results

The investigation leverages five state-of-the-art LLMs (Claude 3.7 Sonnet, GPT-4o, GPT-4o-mini, Gemini 2.0 Flash, DeepSeek V3) on standard public and custom benchmarks (ARC Challenge, MMLU-Pro, GSM8K, MATH, OpenBookQA, NameIndex, MiddleMatch). Prompts for each benchmark item are rewritten in three forms: diluted ($SDE < 0.40$), standard, and ultra-dense ($SDE > 0.80$), strictly preserving semantic intent and answerability.

Strong empirical findings include:

• Ultra-dense prompts (SDE > 0.80) yield a mean accuracy improvement of +8.4 percentage points over diluted prompts, with gains as high as +13.7% on tasks involving lexicon and retrieval (NameIndex).
• No increase in prompt tokens or latency is required for this improvement; ultra-dense prompts are on average 34% shorter than diluted versions.
• Output length (number of generation tokens) is statistically unchanged, evidencing that elevated performance is attributable not to verbosity control but to input informativeness.
• When SDE is combined with IPE, gains are additive, reaching +11.7 percentage points above baseline.

Ablation studies reinforce that SDE, not prompt length minimization or padding, is the causal variable. Controls using length-preserved but semantically neutral prompts yield no improvement, while accuracy correlates monotonically with SDE across a granular spectrum (from 0.20 to 0.80).

Comparison with Existing Prompt Optimization Methods

Earlier prompt engineering methods, such as Chain-of-Thought and Prompt Repetition, generally focus on adding tokens or structuring prompts to enhance sequential processing (e.g., step-wise reasoning or repeated context). Compared to these:

• SDE alone achieves nearly the same accuracy lift (+8.4%) as Chain-of-Thought (+12.1%) but without latency or input length penalty.
• IPE and SDE are additive and orthogonal; Prompt Repetition, by contrast, increases compute cost and only partially addresses attention allocation.
• SDE addresses the intrinsic quality and concentration of information, as opposed to superficial prompt characteristics such as order/length.

Practical Rewriting Heuristics

The authors provide operational heuristics for maximizing SDE:

• Remove all filler, politeness, and metacommentary tokens.
• Replace abstract requests with explicit, concrete, quantifiable instructions (e.g., specifying output format, number of items, units).
• Eliminate redundancy; each piece of unique information appears once.
• Maximize concreteness with specific entities, actions, or quantities.

These recommendations are validated across diverse task typologies (factual, code, analysis, creative), consistently yielding shorter, denser prompts that outperform standard practices, with no observed regression in model output quality.

Implications and Future Directions

The results indicate that semantic density is a key, model-agnostic variable in LLM response quality—one that can be directly manipulated for better accuracy, reduced hallucination, and computational efficiency. Practically, this offers immediate guidelines for API-based LLM integration and can be mechanized to automate prompt rewriting or scoring.

Theoretically, SDE refines the understanding of transformer attention allocation and highlights the non-trivial cost of including low-information context tokens. The findings invite further research into:

• Automated real-time SDE scoring for active prompt optimization.
• Small-scale models ("prompt densifiers") to automatically transform low-density prompts.
• Application to multimodal contexts (text+vision) and long-context windows.
• Domain-specific optimization and fine-tuning regimes that incorporate SDE-maximized data for improved model robustness.

Conclusion

This work provides an operational formalism and experimental evidence for the centrality of semantic density in prompt engineering for LLMs. Maximizing information per token is empirically and theoretically superior to conventional prompt expansions, with zero added overhead and significant, consistent performance gains. SDE optimization emerges as a default recommendation for high-precision, high-efficiency LLM workflows and serves as a foundation for next-generation prompt and context design (2604.17659).