---
title: Less-is-More Hypothesis
url: https://www.emergentmind.com/topics/less-is-more-hypothesis
type: topic
---

# Less-is-More Hypothesis

The Less-is-More Hypothesis refers to the principle that reducing the number or diversity of elements in a system can, under certain conditions, yield improved performance, generalization, interpretability, or efficiency compared to naïvely maximizing quantity. This hypothesis is rigorously supported across domains such as neural network training, data curation, sensor fusion, physics, and social decision-making, where judicious selection, pruning, or filtering of data, model components, or system interactions frequently leads to outcomes that outperform those obtained using all available resources.

## 1. Formal Definitions and Domain-Specific Instantiations

The Less-is-More Hypothesis is not a single mathematical statement, but recurs in problem-specific formulations:

- **Radar-Inertial Odometry (RIO)**: Accurate trajectory estimates are achieved by using strictly fewer radar points, selected via physical properties such as Doppler velocity and radar cross-section. This leads to cleaner, more informative trajectory constraints and demonstrably lower localization error [2402.02200].
- **Instruction Tuning for LLMs**: Subsets of 1k–6k high-quality instructions suffice for strong generalization and style alignment in large models, rivaling performance obtained from ≥50k samples [2311.13133].
- **Data Curation and Learning Theory**: Small, highly curated subsets of training examples, chosen for difficulty and correctness, can yield lower test error than training on the entire set. Theoretical phase transition curves govern when curation improves generalization [2511.03492].
- **Lottery Ticket Hypothesis**: Sparse subnetworks (“winning tickets”) can be found efficiently using only a pruning-aware critical subset of the training data, matching full-data accuracy while massively reducing training iterations [2106.03225].
- **Time Series Forecasting**: Structured pruning of foundation models before task-specific fine-tuning leads to superior accuracy, faster convergence, and improved generalization over full-parameter adaptation [2505.23195].
- **Physical Transport**: In anomalous conductors, increasing the density of scatterers (impurities) can decrease resistance, overturning classical intuitions about additive scattering [2109.08390].

The general principle: quality and relevance of constraints, examples, or parameters dominate sheer quantity.

## 2. Theoretical Foundations and Mathematical Formulations

The hypothesis is substantiated by diverse mathematical frameworks:

- **Data Curation Scaling Laws**: Let $E_{\text{test}}(N,p)$ denote test error with $N$ examples and fraction $p$ retained by a pruning oracle. Closed-form scaling curves reveal when $\min_{q\in Q_p} E_{\text{test}}(N,p) < E_{\text{test}}(N,1)$, i.e., curated subsets outperform the full set given sufficient data abundance and oracle reliability. Analytical phase transitions are observed: for strong generators in the data-rich regime, best performance may arise at aggressive pruning levels ($p \ll 1$) [2511.03492].
- **Few-shot Learning and Entropy**: The Asymptotic Equipartition Property (AEP) implies that almost all probability mass in high-dimensional data lies in a typical set of size $2^{nH}$ (with $H$ entropy rate). Training and generalizing on this set (the “less”) suffices for prediction across the full support (the “more”) [2209.14267].
- **Radar–Inertial Optimization**: The state-space model leverages residual functions derived from physical properties, filtering out ~80% of points as redundant, and casting the estimation as a constrained sliding-window nonlinear optimization with Doppler and cross-section constraints [2402.02200].

## 3. Mechanisms Underlying Less-is-More Effects

Several mechanisms recur in empirical and theoretical investigations:

- **Redundancy Suppression**: Pruning, coreset selection, or coverage optimization removes redundant or low-informative constraints or examples, thus shrinking noise and variance while preserving essential learning signals.
- **Activation of Task-Relevant Substructures**: Pre-training often produces networks with rich, but highly redundant subnets; fine-tuning over a minimal, task-aligned subnetwork regularizes adaptation and improves generalization [2505.23195].
- **Improved Signal-to-Noise Ratio**: Maximum coverage algorithms for synthetic data select diverse, non-overlapping examples, boosting classifier accuracy and data diversity, often at a reduction of ~90% in data volume [2504.14508].
- **Interference and Bottleneck Removal**: In physics, frequent scatterers break up long sub-diffusive bottlenecks, decreasing resistance even as more scatterers are added [2109.08390].

## 4. Empirical Evidence and Quantitative Validation

Rigorous quantitative studies confirm the hypothesis in practice:

| Domain / Task           | Full-data performance | Pruned / Curated subset | Relative data used | References      |
|------------------------|----------------------|------------------------|-------------------|----------------|
| RIO localization       | 0.80m RMSE           | 0.29m RMSE             | 20%               | [2402.02200]   |
| LLM instruction tuning | 0.360 accuracy       | 0.356 (5k subset)      | 8%                | [2311.13133]   |
| Lottery ticket finding | Full test accuracy   | Matched/↑ accuracy     | 35–78%            | [2106.03225]   |
| Synthetic data F1 (SST-2) | 0.80               | 0.81 (500 ACS)         | 10%               | [2504.14508]   |
| TSFM forecasting (ETTm2) | N/A                 | –22.8% MSE             | <50% parameters   | [2505.23195]   |

Experiments reveal sweet spots (e.g. 50% sparsity in CNNs [2302.08878]), benchmarks plateauing in accuracy at small curated subsets, and efficiency gains (92.7% fewer iterations in ticket finding).

## 5. Domain-Specific Applications and Implications

- **Radar-Inertial Sensor Fusion**: Filtering radar points with Doppler and RCS yields trajectory estimates with lower RMSE and computational cost, reflecting that physically validated constraints trump raw data quantity [2402.02200].
- **LLM Instruction Tuning**: Small, high-quality, mixed datasets optimize both factual recall and style alignment, reducing GPU costs and maximizing reproducibility [2311.13133].
- **Data Curation Strategies**: Precise analytic conditions reveal when “less is more” applies—aggressive pruning in abundant data with strong oracles; otherwise, full data is necessary. This framework accounts for contradictory mathematical reasoning strategies (e.g., LIMO vs. Sun et al.) [2511.03492].
- **Synthetic Data for Classifiers**: Adaptive coverage sampling outperforms random or k-means selection, slashing required data and boosting F1-score [2504.14508].
- **Sparse CNN Explainability**: Moderate pruning increases both classifier accuracy and human-rated explainability, linking network sparsity to clearer attribution maps [2302.08878].
- **Human Social Learning**: Minimal social signals (frequency-only) produce higher cumulative payoff than richer rating systems, highlighting behavioral consequences [1406.7560].

## 6. Limitations, Caveats, and Domain Dependency

The Less-is-More effect is not universal. Conditions for its validity include:

- **Generator/Oracle Strength**: Weak generators (or noisy model/data) may require “more is more”—keeping all, or easy, examples.
- **Abundance**: Data-rich regimes support aggressive curation; data-poor scenarios do not.
- **Task Complexity and Representational Completeness**: In reasoning, incomplete pre-training nullifies LIMO benefits [2502.03387]; in TSFMs, over-pruning may disrupt pre-trained priors.
- **Application-specific Constraints**: Excessive pruning can compromise linguistic fidelity in LLMs or degrade classifier accuracy beyond certain thresholds.

## 7. Broader Impact and Future Directions

The hypothesis informs best practices for model fine-tuning, synthetic data pipelines, sensor front ends, and active material engineering. Future work aims to formalize adaptive curation, quantify optimal pruning thresholds, and extend Less-is-More methodologies into new modalities and emerging research areas.

In summary, the Less-is-More Hypothesis provides a cross-disciplinary principle: rigorous selection, filtering, or pruning of constraints, examples, or model components can promote robust learning, efficiency, interpretability, and—paradoxically—sometimes even yield strictly better performance than indiscriminate maximization of quantity. Its domain-specific realizations are governed by well-defined mathematical and empirical conditions [2402.02200][2311.13133][2511.03492][2106.03225][2505.23195][2504.14508][2302.08878][2109.08390][1406.7560][2209.14267][2502.03387].

Source: https://www.emergentmind.com/topics/less-is-more-hypothesis