SP-Random Walk Foraging Model

• Foraging Model is a framework that uses reinforcement learning to tune movement parameters for optimal search efficiency in complex, partially observed landscapes.
• It learns adaptive step-length distributions via stochastic policy learning, offering a flexible alternative to static models like Lévy walks.
• Empirical evaluations reveal that RL-trained policies can exceed traditional methods by up to 10% in target detection efficiency.

SP-Random Walk

A SP-Random Walk ("Stochastic-Policy Random Walk," Editor's term) is a movement and learning paradigm in which a foraging agent, biological or artificial, discovers adaptive step-length strategies via stochastic policy learning, typically by reinforcement learning (RL) or Markov Chain Monte Carlo (MCMC) in foraging, search, or exploration environments. Unlike models that pre-specify a search-step distribution (e.g., Lévy walks), the SP-Random Walk framework allows the agent to tune or learn movement parameters to maximize foraging efficiency in complex, often partially observed landscapes. This approach fundamentally connects optimal foraging theory, stochastic movement ecology, and the learnability of efficient behaviors in high-dimensional, uncertain settings (Muñoz-Gil et al., 2023).

1. Formal Definition and Mathematical Framework

Let $S_n$ be the "state" of the agent at small step $n$, typically a scalar step counter, memory vector, or local environmental observation. Let $A$ denote the set of actions, with the minimal movement action space $A = \{\uparrow, \curvearrowleft\}$: continue or turn. The stochastic policy $\pi(a|S_n)$ governs the probability of action $a$ at state $S_n$: $\Pr(a_n = a \mid S_n) = \pi(a|S_n)$ For a given episodic or continuous trajectory, the step-length distribution $\Pr(L=Nd)$ generated by this policy is: $$\Pr(L=Nd) = \pi(\curvearrowleft|N) \prod_{n=1}^{N-1} \pi(\uparrow|n)$$ where $L$ is the step length (in discrete steps of size $d$), and $\pi(\curvearrowleft|N)$ is the probability of turning after $N$ straight steps (Muñoz-Gil et al., 2023).

The agent's cumulative reward $R_T$ over time horizon $T$ is directly proportional to the number of target detections in non-destructive search: $\eta = \lim_{T \rightarrow \infty} \frac{n_T}{T}$ where $n_T$ is the total number of unique detections—this is the classical foraging efficiency (Muñoz-Gil et al., 2023).

In the RL optimization, the connection between the stochastic policy and efficiency is direct: maximizing the average reward per step,

$$R_\pi = \lim_{T \to \infty} \frac{1}{T} \mathbb{E}_\pi \left[ \sum_{t=0}^T R(e_t,a_t) \right]$$

recovers optimal foraging efficiency: $\eta = R_\pi$ This establishes the formal equivalence between the stochastic search policy learning and foraging optimization (Muñoz-Gil et al., 2023).

2. RL-Based Learning of Step Distributions

The core of the SP-Random Walk is policy optimization via RL or analogous Bayesian policy sampling. In the model of Muñoz-Gil et al., the agent's state is the scalar counter $n$ since its last turn; actions are "continue" or "turn." Projective Simulation (PS)—a variant of policy-gradient RL—updates a bipartite memory graph with edge values $h(s,a)$ for each state-action pair. Policy probabilities are computed: $$\pi(a|s) = \frac{h(s,a)}{h(s, \uparrow) + h(s, \curvearrowleft)}$$ After each reward, the policy is updated along traversed ("glowed") edges. No explicit $\epsilon$-greedy noise is needed, as stochasticity arises intrinsically from the learned weights (Muñoz-Gil et al., 2023).

Hyperparameters include initialization $$h_0(s, \uparrow)=0.99,\ h_0(s, \curvearrowleft)=0.01$$ (bias toward long steps), glow decay, and forgetting rates. Policies are evaluated post-episode over $10^4$ full walks per agent (Muñoz-Gil et al., 2023).

3. Empirical Findings and Efficiency Comparisons

RL-trained SP-Random Walk policies consistently match or surpass search efficiencies achieved by the best fixed-form hypotheses, including Lévy and optimized bi-exponential step distributions (see Table 1, which compactly summarizes observed outcomes):

Policy Class	Main Shape	Relative Efficiency
Lévy (β optimized)	Power-law tail, single mode	Baseline, reference
Bi-exponential	Bimodal (two characteristic scales)	Sometimes superior to Lévy
RL-learned	Peak at $l_c$, fat tail	Exceeds both by up to 10%

The RL agent's optimized $\pi(\curvearrowleft|n)$ typically produces:

• A sharp peak at $n \approx l_c$ (the environmental "reset" or teleport displacement).
• A plateau or decaying tail for larger $n$, adapting to the mean inter-target distance $\sim 1/\sqrt{\rho}$ ($\rho$ being the spatial target density).

This yields a realized step-length distribution with two characteristic scales—outperforming mono-modal or simple two-component mixtures (Muñoz-Gil et al., 2023).

4. Mechanistic Insights: Why SP-Random Walk Outperforms Fixed Ansätze

Key reasons for superior efficiency:

• The agent learns to "turn at $l_c$," exploiting the geometry of resets to avoid already-explored regions.
• The learned policy flexibly adapts to combine more than two step scales if needed, optimizing exploration in multi-scale environments.
• No parametric step-length distribution constrains the solution; RL finds forms matched to ecological or energetic landscape features.
• The mapping between policy and step distribution is invertible, so any target distribution implementable by independent steps can, in principle, be learned (Muñoz-Gil et al., 2023).

5. Environmental and Model Generalizations

SP-Random Walks generalize to multiple settings:

• Heterogeneous and nonuniform target fields, obstacles, or gradients (no change in algorithmic principle).
• Multi-agent extensions, with swarm behaviors implemented by applying RL to each agent, possibly with communication or social cues.
• Memory-augmented (non-Markovian) versions, in which state $S_n$ includes additional history for implementing learning over longer time horizons (Muñoz-Gil et al., 2023).
• Applications to robotics, search-and-rescue operations, scalable network or database search under environmental refresh (Muñoz-Gil et al., 2023).

6. Implications for Optimality and Learnability in Foraging

SP-Random Walk formalizes the reconciliation between optimality (existence of maximally efficient movement distributions) and learnability (the agent's capacity to discover such distributions iteratively by reward-based trial and error), addressing a historic debate in evolutionary ecology. The established equivalence between steady-state RL reward and classical search efficiency $\eta$ implies that foraging efficiency need not depend on pre-evolved or hardwired movement heuristics; rather, optimal strategies may emerge from general-purpose learning mechanisms (Muñoz-Gil et al., 2023).

7. Broader Theoretical Context and Limitations

SP-Random Walk approaches encompass and extend prior random-walk foraging models:

• Lévy or power-law random walks are recovered as special cases when such distributions are optimal in a given context (Zhao et al., 2014).
• SP-Random Walks admit complex movement statistics without requiring hand-tuning or external optimization.
• RL policy learning is robust to environmental change, provided sufficient exploration.

Key limitations include:

• Policy learning may require long training times or environmental stationarity for convergence.
• If real animals are constrained by stronger computational or sensory limits, realizable SP-Random Walk policies may be only a subset of those permitted by the model.
• Empirical verification in animal movement data must account for partial observability and other confounds.

In sum, SP-Random Walks represent a principled, mechanistic framework for the endogenous discovery and execution of optimal movement strategies in foraging and search, grounded in explicit connections between stochastic policy learning, reward maximization, and classical search efficiency (Muñoz-Gil et al., 2023).