Neural Thickets in Pretrained Models

• Neural thickets are dense clusters of task-specific specialists around a pretrained weight vector, offering abundant optimal perturbations for diverse downstream tasks.
• The RandOpt method employs parallel random sampling and ensemble selection to efficiently harness these specialists, outperforming traditional fine-tuning in several domains.
• Scaling laws reveal that as model size increases, the solution density rises, reducing reliance on iterative gradient descent and enhancing model adaptation.

Neural thickets refer to the regime in which large, well-pretrained neural networks attain a parameter space characterized by a high density of diverse task-improving specialists clustered in the local neighborhood around the pretrained weights. Rather than presenting a unique solution vector, a pretrained model of sufficient scale is surrounded by a dense set of task experts—parameter vectors which, with small perturbations from the base weights, exhibit superior performance on downstream tasks. This stands in contrast to the “needle-in-a-haystack” property found in small or untrained models, where such task-expert solutions are exceedingly rare and necessitate carefully structured optimization to locate them (Gan et al., 12 Mar 2026).

1. Pretrained-Weights Distribution and Solution Density

Pretraining yields a parameter vector $\theta_0$, typically regarded as a starting point for downstream optimization. In the thicket regime, $\theta_0$ is viewed as the mean of a local isotropic Gaussian distribution over weights: $P(\theta) = \mathcal{N}(\theta_0, \sigma^2 I_n)$, where $n$ is the number of model parameters and $\sigma$ is a hyperparameter setting the exploration radius. For a performance metric $s: \mathbb{R}^n \rightarrow \mathbb{R}$, the solution density at margin $m\geq0$ is defined as

$$\delta(m) = \mathbb{P}_{\theta \sim \mathcal{N}(\theta_0,\sigma^2 I)}[s(\theta) \geq s(\theta_0)+m] = \mathbb{E}_{\epsilon \sim \mathcal{N}(0,\sigma^2 I)}\left[1_{\,s(\theta_0+\epsilon)\geq s(\theta_0)+m}\right].$$

This can be interpreted as the fraction of volume within a Euclidean ball of radius $r$ about $\theta_0$ containing parameter vectors with at least $\theta_0$0-margin improvement in performance. Empirical estimates of $\theta_0$1 are obtained via Monte Carlo sampling with typical settings (e.g., $\theta_0$2) (Gan et al., 12 Mar 2026).

2. Random Guessing and Ensembling (RandOpt)

RandOpt is a fully parallel, two-phase post-training procedure for exploiting neural thickets. First, $\theta_0$3 seeds are sampled and assigned a noise scale $\theta_0$4 (a discrete set). Each seed produces a perturbed model $\theta_0$5, $\theta_0$6, whose performance $\theta_0$7 is then evaluated on a small validation set. The $\theta_0$8 best-performing perturbations are retained: $\theta_0$9 Ensemble inference aggregates the predictions of these $P(\theta) = \mathcal{N}(\theta_0, \sigma^2 I_n)$0 models via majority vote for each test input. This approach, implemented by a simple pseudocode loop over seeds and ensemble majority vote, operates entirely in parallel and with a single round of communication to select the top-$P(\theta) = \mathcal{N}(\theta_0, \sigma^2 I_n)$1 (Gan et al., 12 Mar 2026).

3. Scaling Laws and the Thicket Regime

Empirically, the solution density $P(\theta) = \mathcal{N}(\theta_0, \sigma^2 I_n)$2 increases nontrivially with model size. For small or untrained models, $P(\theta) = \mathcal{N}(\theta_0, \sigma^2 I_n)$3 for $P(\theta) = \mathcal{N}(\theta_0, \sigma^2 I_n)$4, indicating a “needle” regime. For large, well-pretrained models, $P(\theta) = \mathcal{N}(\theta_0, \sigma^2 I_n)$5 grows monotonically (often exceeding 20–60% for $P(\theta) = \mathcal{N}(\theta_0, \sigma^2 I_n)$6), marking the emergence of thickets. The observed scaling law is characterized as $P(\theta) = \mathcal{N}(\theta_0, \sigma^2 I_n)$7 for some $P(\theta) = \mathcal{N}(\theta_0, \sigma^2 I_n)$8 until saturation. For example, Qwen2.5-0.5B yields $P(\theta) = \mathcal{N}(\theta_0, \sigma^2 I_n)$9, while Qwen2.5-32B achieves $n$0 for GSM8K (Gan et al., 12 Mar 2026).

4. Task Specialization and Spectral Discordance

The diversity of task specialists in neural thickets is quantified using a percentile-rank matrix $n$1 (for $n$2 seeds and $n$3 tasks) and the Pearson correlation matrix $n$4. Spectral discordance $n$5 measures the degree of orthogonality among top-performing specialists:

• $n$6 implies generalist seeds;
• $n$7 near maximal ($n$8) indicates maximal specialization.

Empirically, with increasing model size, $n$9 rises (from near 0 at 0.5B to $\sigma$0 at 32B for $\sigma$1), and PCA analyses demonstrate distinct clustering of specialist seeds (Gan et al., 12 Mar 2026).

5. Empirical Demonstration Across Modalities

RandOpt, with $\sigma$2, $\sigma$3, matches or outperforms sequentially optimized baselines such as PPO, GRPO, and ES under matched FLOPs budgets:

• On Qwen2.5-1.5B-Inst (Countdown): Base: 6.7%, PPO: 27.0%, GRPO: 27.5%, ES: 44.2%, RandOpt: 52.7%.
• On Qwen2.5-3B-Inst (GSM8K): Base: 79.8%, PPO: 83.1%, GRPO: 83.2%, ES: 85.8%, RandOpt: 87.1%.

Extensions to other domains include:

• Vision-LLMs (e.g., Qwen2.5-VL-3B-Inst for GQA): accuracy improved from 56.6% (base) to 69.0% (RandOpt).
• Diffusion models (Stable Diffusion XL): parameter perturbations yield “color thickets.” Ensemble selection enables explicit style control via simple criteria such as color-score (Gan et al., 12 Mar 2026).

6. Implications for Model Adaptation and Optimization

The thicket regime suggests that pretrained models constitute distributions over weight space rather than single optima. In large-scale models, random search combined with ensembling provides effective post-training adaptation—minimizing the necessity for iterative gradient descent. RandOpt’s strictly parallel nature, requiring only a single round of score communication, is advantageous in federated or high-latency environments. The additional computational overhead of ensemble inference can be reduced by distillation: supervised fine-tuning on top-$\sigma$4 ensemble traces recovers most performance with less than 2% additional FLOPs. The density of task experts in thickets also indicates a MAML-like property, with pretrained models positioned such that many task-specific minima are a single step away (Gan et al., 12 Mar 2026).

7. Broader Context and Conclusion

Neural thickets alter the conceptualization of model adaptation in overparameterized, well-pretrained architectures. Once pretraining attains sufficiency, the region surrounding $\sigma$5 is not sparse with respect to downstream task experts but densely packed—rendering parallel random sampling and majority-vote ensembling a competitive alternative to conventional fine-tuning. This reinterprets the pretrained weight space as a locus of high-probability, diverse, task-orthogonal solutions, providing new algorithmic and theoretical lenses for the study and exploitation of large model landscapes (Gan et al., 12 Mar 2026).