Entropy-Driven Drift as a Source of Optimization Difficulty in Combinatorial Spaces

Abstract: Understanding the origin of optimization difficulty in high-dimensional combinatorial spaces remains a fundamental problem. Existing perspectives typically characterize difficulty in terms of properties of states, their connectivity, or distributions over states. However, search algorithms operate as stochastic processes evolving over time, and optimization is inherently a trajectory-level phenomenon. This motivates a shift from state-based to trajectory-based analysis. In this work, we adopt a trajectory-based perspective and analyze search dynamics through the evolution of a distance process. We identify a structural mechanism, which we term entropy-driven drift. This mechanism systematically biases trajectories toward high-entropy regions. This drift arises from asymmetry in local transitions induced by the underlying graph structure, independent of objective variation. In the absence of objective variation, trajectories that reach the target are atypical under the induced dynamics, leading to a discrepancy between rapid mixing and slow hitting. We formalize this mechanism in a canonical combinatorial setting with a highly symmetric underlying graph, where the symmetry allows explicit characterization of the induced drift. The mechanism highlights entropy-driven drift as a source of optimization difficulty and provides a trajectory-level framework for understanding search dynamics in combinatorial spaces.

• The paper demonstrates that entropy-driven drift steers random walk trajectories away from target configurations in high-dimensional combinatorial spaces.
• It provides an explicit formulation for the drift in Johnson graphs, revealing an equilibrium distance that aligns with the shell of maximum entropy.
• The study highlights that conventional local search and energy-driven methods are insufficient to overcome entropy barriers, suggesting the need for non-local or population-based approaches.

Entropy-Driven Drift as a Source of Optimization Difficulty in Combinatorial Spaces

Introduction and Problem Formulation

This paper offers a trajectory-based analysis of combinatorial optimization difficulties, focusing on the phenomenon termed entropy-driven drift. Unlike classical perspectives framing optimization complexity in terms of the size or connectivity of the feasible set, or state-level measures such as fitness landscapes and fitness-distance correlation (FDC), this work investigates how the evolution of stochastic search processes is systematically biased by combinatorial structure. Specifically, it is shown that in high-dimensional $k$-of-$n$ subset selection spaces (Johnson graphs), the prevalence of high-entropy regions induces an intrinsic drift that actively steers trajectories away from the target, even in the absence of objective-driven gradients.

Structural and Trajectory Analysis in Johnson Graphs

The analysis centers on the Johnson graph $J(n,k)$, a canonical model for $k$-element subset selection. The feasible space is naturally partitioned into distance shells around a reference (target) configuration. Shell cardinalities, given by $|\Omega_d|=\binom{k}{d}\binom{n-k}{d}$, aggregate into a unimodal distribution, peaking at an intermediate distance rather than near the target. This combinatorial structure reflects a pronounced entropy landscape over the distance coordinate $d$.

Figure 1: Sample trajectories of the distance process in random walks on $J(200,40)$ (starting at $d=1$) illustrating systematic drift toward the entropy equilibrium $d^*=32$.

Most configurations reside far from any fixed target—a typical concentration of measure phenomenon in high dimensions. Consequently, under local search dynamics (random swaps), transition probabilities are heavily asymmetric: moves away from the target are combinatorially favored, while moves toward the target are exponentially suppressed at the level of aggregate edge multiplicities.

Quantitative Characterization of Entropy-Driven Drift

An explicit formulation for the drift of the distance process $Y_t$ under random walks in $n$0 is derived:

$n$1

The drift vanishes at a unique equilibrium $n$2, which coincides with the region of maximal shell entropy in the thermodynamic limit.

Figure 2: Entropy landscape $n$3, entropy increment $n$4, and drift $n$5 on $n$6.

Beneath this, the ratio of probabilities for inward vs. outward moves is governed by the ratio of shell cardinalities, translating directly into entropy gradients—a trajectory in low-entropy regions (near the target) experiences strong outward drift.

The expected hitting time to the target from distance $n$7 is also explicitly computed, revealing exponential scaling in $n$8 for typical regimes. Thus, even though the random walk mixes rapidly over the state space, target-reaching events are exponentially rare, reflecting a sharp divergence between mixing and hitting times.

Interplay with Objective-Driven Guidance

The analysis is extended to scenarios where an idealized objective function provides "energy" gradients toward the target, and its influence is tuned via a Metropolis acceptance criterion ($n$9 parameter). The net drift becomes

$J(n,k)$0

where $J(n,k)$1 are local outward/inward move probabilities. Notably, unless $J(n,k)$2 is very large, entropy-driven drift remains dominant near the target, establishing a persistent entropy barrier even with strong objective signals.

Figure 3: Drift under Metropolis dynamics for various $J(n,k)$3; entropy-driven drift remains positive near $J(n,k)$4 unless $J(n,k)$5 is large.

Empirically, with finite $J(n,k)$6, search trajectories stabilize around nonzero equilibrium distances $J(n,k)$7 rather than concentrating at the target.

Figure 4: Sample trajectories under idealized objective guidance ($J(n,k)$8) remain away from $J(n,k)$9, stabilizing at $k$0.

The expected hitting time is sharply reduced as $k$1 increases, but local transition asymmetry still impedes efficient discovery of the target unless exploration is essentially greedy (which in real problems, creates susceptibility to local energy minima).

Figure 5: Expected hitting time to target vs. initial distance, for multiple $k$2 values; substantial acceleration only when $k$3 is large.

Comparison with IID Uniform Sampling

A comparison to IID uniform sampling (i.e., choosing random configurations globally) reveals that the random walk's expected hitting time is consistently greater (except in trivial proximity to the target), quantifying the cost imposed by locality and entropy-driven drift.

Figure 6: Log ratio of expected hitting times (random walk vs. IID) indicates that local dynamics are typically less efficient due to entropy-induced barriers.

Theoretical and Practical Implications

This work reaffirms that optimization challenge in high-dimensional combinatorial settings is fundamentally a trajectory-level phenomenon. The entropy-driven drift, rooted in the structural asymmetry of local transitions, cannot be explained by static, state-based measures alone, nor is it eliminated by conventional objective-based or local search heuristics. This mechanism is robust under minor variations in problem structure and extends beyond $k$4 to any setting where the number of configurations varies non-uniformly with graph distance.

Critical implications include:

• Distinction between entropy and energy barriers: Even flat objective landscapes are characterized by large entropy barriers that make target-reaching trajectories rare.
• Local search limitations: Purely local or energy-driven methods (e.g., simulated annealing with moderate $k$5) cannot fully neutralize entropy-driven drift, resulting in trajectory distributions concentrated away from the target.
• Role of transition structure: Only algorithms that modify the underlying graph (e.g., non-local jumps, global crossover, or population-based search) are capable of altering or circumventing entropy-driven drift, justifying the empirical efficiency of such approaches in some combinatorial optimization problems.
• Information limitations: Since the coordinate governing the drift (distance to an unknown target) is unobservable to the algorithm, explicit compensation for entropy bias is usually infeasible unless additional target information is available.

The framework clarifies why rapid mixing is not equivalent to rapid solution discovery, explains the empirical challenges of local search in high dimensions, and gives a rigorous foundation for large-deviation perspectives of optimization.

Conclusion

The trajectory-based analysis of random walks in combinatorial spaces demonstrates that entropy-driven drift, induced by the interaction of state space structure and local transitions, is a structural source of optimization difficulty. This mechanism operates independently of objective landscape ruggedness, is robust to variations in target location and multiplicity, and highlights a general and underappreciated barrier in local search optimization.

Approaches that only modify transition weighting (without altering transition structure) can modulate but not eliminate entropy-driven drift. Overcoming this phenomenon necessitates either fundamentally different search operators, population-based approaches, or algorithmic frameworks with access to additional information (e.g., those that adaptively estimate or exploit latent drift structure).

This trajectory perspective invites further theoretical development connecting combinatorial optimization, large deviations theory, and controlled stochastic processes, particularly for the quantification and manipulation of non-typical trajectory distributions in high-dimensional search spaces (2604.17332).