Less-Is-More Hypothesis Overview

• Less-Is-More Hypothesis defines that using fewer, critically selected resources can equal or exceed performance in complex systems.
• Empirical studies in machine learning, neuroscience, and physics demonstrate that strategic constraints, such as data pruning and energy limits, improve efficiency and robustness.
• The hypothesis underpins methodologies like sparse coding, optimization under constraints, and prompt engineering, revealing non-monotonic performance dynamics.

The "Less-Is-More Hypothesis" (or LIMH) formulates a recurring principle observed across a wide range of scientific domains: in certain complex systems—biological, physical, computational, and sociotechnical—deliberately using fewer resources, less data, less structure, or less information, or imposing strategic constraints, can yield equal or even superior performance compared to employing more. This effect is generally non-monotonic: beyond a certain point, increasing quantity or complexity harms efficiency, generalization, or robustness. Within machine learning, physics, neuroscience, language engineering, and collective behavior, the hypothesis is instantiated via formal resource-constrained optimization, sparse coding, data pruning, bandit feedback granularity, activity percolation, and prompt engineering. The LIMH provides both algorithmic recipes and theoretical rationales for maximizing practical performance with fewer, higher-quality, or more critically selected ingredients.

1. Formal Definitions and General Frameworks

Several mathematical formalizations of the Less-Is-More hypothesis have been advanced. In statistical learning and signal processing, LIMH can be conceptualized in terms of entropy, bandwidth, or effective sample complexity:

• Optimization under resource constraints: For a learner defined by parameter vector $\theta$ and performance $P(\theta)$, impose explicit bounds on bandwidth $C(\theta)$ and energy $E(\theta)$:

$$\max_\theta P(\theta) \quad \text{s.t. } C(\theta) \leq B,\, E(\theta) \leq E_{\max}$$

The "less" refers to tight quota $B$, $E_{\max}$; the "more" is observed in efficient, robust solutions that emerge in these tight regimes (Cohen et al., 8 Jun 2025).

• Sparse/critical set selection: For data $\mathcal{D}_0$, define a minimal "Pruning-Aware Critical set" (PrAC) $\mathcal{P} \subset \mathcal{D}_0$ as the union of high-forgetting and pruning-difficult examples (Zhang et al., 2021). The key claim: winning lottery tickets, i.e., performant sparse subnetworks, can be found using only $\mathcal{P}$, with $P(\theta)$0.
• Information-theoretic sampling: The Minimal Sufficient Typical set $P(\theta)$1 from the Asymptotic Equipartition Property specifies the smallest sample set required for generalization error below a fixed threshold, yielding training set sizes $P(\theta)$2 (Pereg et al., 2022).
• Prompt cognitive load: In LLM reasoning, prompts exceeding a model's finite capacity (in bytes or tokens) can saturate or even reduce accuracy due to increased cognitive load; thus, brevity and optimal ordering can outperform exhaustive instruction sets (Cazares, 20 Apr 2026).

The LIMH is not a universal law. Rather, it is an emergent property of systems with resource constraints, over-parameterization, redundant representations, or robust critical subsets. The effect is context-dependent and must be established via rigorous empirical or theoretical analysis.

2. Empirical Validation Across Domains

The LIMH has been instantiated and verified in diverse fields. Key empirical demonstrations include:

• Deep Learning and Sparsity: By training and pruning only on a PrAC set—constituting 35.32–78.19% of the full data in image tasks—researchers locate high-quality sparse subnetworks (lottery tickets) while saving 63.54–92.77% of training iterations, with no accuracy degradation (Zhang et al., 2021).
• Time Series Forecasting: Aggressive structured pruning of large Time Series Foundation Models (TSFMs) followed by targeted fine-tuning improves mean squared error by up to 22.8% relative to full model fine-tuning, despite updating far fewer parameters (Zhao et al., 29 May 2025).
• Instruction Tuning in LLMs: Randomly chosen 1–6K instruction-response pairs (~1.6–10% of large benchmarks) suffice to match or exceed the benefits of finetuning on datasets an order of magnitude larger, particularly when mixing "assistant-style" and "benchmark-style" demonstrations (Jha et al., 2023).
• Mathematical Reasoning and Prompt Engineering: Carefully crafted, concise prompts (289–2,252 B) for LLMs lead to peak accuracy plateaus (60–79%) on formal logic tasks, while longer, more complex prompts decrease performance due to capacity limitations (Cazares, 20 Apr 2026).
• Human Collective Behavior: In multi-armed bandit games, groups provided only coarse frequency signals (vs. fine-grained 5-star ratings) collectively achieve higher rewards; over-informative feedback induces herding and free-riding, reducing payoff (Toyokawa et al., 2014).
• Physical Transport: In subdiffusive quantum spin chains, adding more impurities can paradoxically enhance transport (increase diffusion constant): $P(\theta)$3, with $P(\theta)$4 and $P(\theta)$5, so $P(\theta)$6 increases as impurity density $P(\theta)$7 increases (Znidaric, 2021).
• Active Solids and Odd Elasticity: In unpercolated non-reciprocal lattices, increased microscopic activity parameter $P(\theta)$8 drives the macroscopic odd modulus $P(\theta)$9 to zero for large $C(\theta)$0, i.e., "more is less" in the activation response (Binysh et al., 25 Apr 2025).

3. Mechanistic Insights and Theoretical Explanations

The mechanisms underlying LIMH are varied but converge on several themes:

• Selective Criticality: Systems often contain a small subset of "critical" entities (data points, subnetworks, active sites) that disproportionately determine global behavior or generalization. Examples include hard-to-memorize or highly-forgotten data points in neural modeling (Zhang et al., 2021), subnetwork priors in TSFMs (Zhao et al., 29 May 2025), and finite-size percolated clusters in active solids (Binysh et al., 25 Apr 2025).
• Overfitting and Cognitive Overload: Excess complexity, whether in model size, prompt engineering, or social information, can saturate or degrade performance. Transformer LLMs display non-monotonic accuracy as prompt size grows, with significant drops past 2KB for certain models (Cazares, 20 Apr 2026). Fine informational granularity (e.g., star ratings) induces social herding, impeding exploration and cumulative reward (Toyokawa et al., 2014).
• Implicit Regularization by Constraint: Explicit bandwidth, energy, or sparsity constraints can drive emergent symbolic codes, parsimony, and robustness, as seen in spiking neurons and reservoir computing, where severely limited resources yield near-state-of-the-art generalization (Cohen et al., 8 Jun 2025).
• Nonlinear Resource–Performance Dynamics: In some physical systems, transport or macroscopic response is a non-monotonic function of disorder or activity. Above critical thresholds (percolation, anomalous exponents) additional microscopic resources (impurities, activity) can suppress rather than enhance the large-scale effect (Znidaric, 2021, Binysh et al., 25 Apr 2025).
• Cognitive Templating and Knowledge Elicitation: For large foundation models, minimal sets of high-quality reasoning templates (e.g., 800 curated demonstrations) suffice to elicit advanced out-of-distribution reasoning, conditional on pretraining completeness (Ye et al., 5 Feb 2025).

4. Methodologies and Algorithmic Templates

Several algorithmic schemas instantiate LIMH:

• Pruning-Aware Critical Set Construction: Alternate between computing sample forgetting statistics and pruning network weights to isolate hard examples and disagreement cases, continually trimming data and model in tandem (Zhang et al., 2021).
• Loss-Guided Structured Pruning: Compute per-channel loss sensitivity via gradients and the Fisher information; prune iteratively and batch-wise to reach target sparsity before final fine-tuning (Zhao et al., 29 May 2025).
• Reduced-Entropy Few-Shot Learning: Utilize receptive-field normalization to eliminate input redundancy before processing by minimal RNNs trained on the typical set. Exploit AEP-derived scaling in chosen training set size (Pereg et al., 2022).
• Prompt Engineering with Cognitive Load Constraints: Iteratively test prompt variants of increasing length, heuristic coverage, and ordering; identify the empirical single-prompt ceiling and proceed to multi-prompt or interactive strategies when surpassed (Cazares, 20 Apr 2026).
• Coarse-to-Fine Data Filtering: Deploy macro-screening (e.g., logistic regression on LLM-extracted features) and micro-level semantic segmentation (sliding-memory chunking) to extract decision-critical tokens for SFT, as in the STITCH framework for agentic LLMs (Team et al., 1 Apr 2026).
• Optimization Pass Sequencing: Generate all prefix sub-configurations of compiler optimization pipelines, empirically selecting the shortest one exceeding the baseline (e.g., LLVM -O2) in runtime or energy (Georgiou et al., 2018).

5. Impact, Applications, and Generalization

The LIMH yields operational gains in compute efficiency, data requirements, transferability, and—in several settings—absolute performance:

• Resource and Time Savings: Across deep learning, pruning and critical subset selection reduce data and computation by over an order of magnitude while matching baseline accuracy (Zhang et al., 2021, Zhao et al., 29 May 2025).
• Transferability: Pruned critical sets and subnetworks generalize across architectures, amortizing selection costs (Zhang et al., 2021).
• Regularization and Robustness: When combined with explicit constraints, systems manifest improved generalization, resilience to overfitting, and rapid learning from sparse data, as seen in biological and developmental studies (Cohen et al., 8 Jun 2025).
• System Design: Insights from LIMH inform how compiling, tuning, or architecting hardware/software stacks can favor early-stopping, coarse granularity, or minimal sufficient templates for optimal performance (Georgiou et al., 2018, Jha et al., 2023).
• Physics and Materials: Nonlinear transport and mechanical response in many-body physics directly instantiate LIMH, especially in disordered or sub-critical regimes (Znidaric, 2021, Binysh et al., 25 Apr 2025).
• Agentic and Reasoning LLMs: Curated, high-value trajectories enable amplified code-generation and agentic behavior with minimal training cost, supporting advances in software engineering agents (Team et al., 1 Apr 2026).

6. Limitations and Open Questions

Despite strong empirical and theoretical backing, the scope and universality of LIMH remain qualified:

• Domain and Architecture Dependence: For extreme distribution shifts, insufficient pretraining, or highly unconstrained models, the reductionist LIMH approach can fail or show diminishing returns (Ye et al., 5 Feb 2025).
• Capacity and Model Saturation: In prompt engineering, the effect saturates at model-specific limits, requiring ensemble or hierarchical strategies to break ceilings (Cazares, 20 Apr 2026).
• Quantifying Criticality: Automatically identifying critical data or model subsets is nontrivial, often computationally intensive, especially for high-dimensional, non-stationary domains (Zhang et al., 2021, Zhao et al., 29 May 2025).
• Evaluation Paradigm Bias: In LLMs, reliance on automated judges (e.g., GPT-4) for qualitative assessment presents bias and reproducibility challenges (Jha et al., 2023).
• Physical Realizability: In physics, system parameters (e.g., percolation threshold, anomalous exponents) must be precisely tuned for LIMH to emerge, and the regime may be fragile to perturbations (Binysh et al., 25 Apr 2025).

Open directions include developing adaptive filtering methods for data/model selection, exploring the limits of LIMH in symbolic versus neural domains, extending "less-is-more" principles into active, embodied real-world systems (Cohen et al., 8 Jun 2025), and unifying formal characterizations of criticality and redundancy across disciplines.

7. Synthesis and Design Principles

LIMH-driven research increasingly points toward a set of design implications:

• Impose and embrace parsimony: Explicitly constrain energy, bandwidth, or sample budgets to coax efficient codes and learning.
• Select or filter for criticality: Prefer data/model points with high information density, loss sensitivity, or "hardness," whether via statistical, information-theoretic, or algorithmic criteria.
• Favor co-design: Combine data and model sparsity for synergistic reductions in resource usage and improvements in generalization (Zhang et al., 2021).
• Limit information granularity: In sociotechnical systems, coarser, less detailed feedback often prevents harmful herding or stagnation (Toyokawa et al., 2014).
• Optimize for robust generalization, not mere capacity: Resource constraints often function as implicit regularizers, inducing codes and dynamics well suited for transfer and adaptation (Cohen et al., 8 Jun 2025, Pereg et al., 2022).
• Guard against over-complexification: Monitor for cognitive or computational overload in prompts, architectures, or interactions, adopting multi-stage or decomposed strategies as needed (Cazares, 20 Apr 2026).

The LIMH continues to challenge the prevailing maxim, "more is better," demonstrating across theory and application that, when judiciously selected, "less" can indeed—often and robustly—be more.