---
title: 'HiFo-Prompt: LLM Heuristic Design'
url: https://www.emergentmind.com/topics/hifo-prompt
type: topic
---

# HiFo-Prompt: LLM Heuristic Design

Searching arXiv for the primary paper and closely related prompt-optimization work to ground the article in current literature.
HiFo-Prompt is a framework for **LLM-based automatic heuristic design** within **evolutionary computation**, built around two complementary prompting mechanisms: **Foresight** and **Hindsight**. Rather than optimizing a natural-language prompt for a downstream language task, it uses prompts to steer an LLM that generates executable heuristics—typically Python code plus a brief natural-language “thought”—inside an evolutionary search loop. Foresight adaptively regulates exploration and exploitation from population dynamics, while Hindsight distills successful heuristics from earlier generations into reusable design principles stored in a persistent knowledge base. Across traveling salesman, online bin packing, flow shop scheduling, and cost-aware Bayesian optimization, the method is reported to produce stronger heuristics, faster convergence, and better query efficiency than prior LLM-based automatic heuristic design pipelines [2508.13333].

## 1. Scope and disambiguation

HiFo-Prompt refers specifically to **"HiFo-Prompt: Prompting with Hindsight and Foresight for LLM-based Automatic Heuristic Design"** [2508.13333]. Its domain is not general-purpose prompt engineering for text classification, reasoning, or instruction following, but the generation and evolution of **problem-specific heuristic programs** inside an EC workflow. The prompt is therefore an internal control signal for heuristic synthesis, rather than the final artifact evaluated on a conventional NLP benchmark.

This distinguishes HiFo-Prompt from several nearby but different research lines. It is not **HiPrompt**, the higher-resolution image generation method based on hierarchical MLLM prompts and frequency-aware denoising [2409.02919]. It is also distinct from prompt optimizers that search over discrete natural-language prompts for black-box or in-box LLM tasks, such as APOHF [2405.17346], HbBoPs [2412.07820], FIPO [2402.11811], GFlowPO [2602.03358], OPTS [2503.01163], HAPO [2601.02683], and DelvePO [2510.18257]. Nor is it an instruction-alignment framework like HIPO, which formalizes system–user priority as constrained policy optimization [2603.16152]. HiFo-Prompt belongs instead to the literature on **automatic heuristic design**, where LLMs are used as semantic generators of algorithmic operators and search policies [2508.13333].

## 2. Formal problem setting

The paper formulates automatic heuristic design as search over a heuristic space $\mathcal{H}$, with the optimal heuristic defined as
$$
h^* = \mathop{\arg\max}_{h \in \mathcal{H}} g(h),
$$
where
$$
g(h) = \mathbb{E}_{\mathrm{ins} \in D} \left[ -f(h(\mathrm{ins})) \right].
$$
Here $f$ is the underlying task objective to minimize, so maximizing $g(h)$ amounts to finding heuristics with lower expected objective value on an instance distribution $D$ [2508.13333].

The evolutionary abstraction used in the paper writes population dynamics as
$$
P^{(t+1)} \sim \mathcal{F}(P^{(t)} \mid \theta^{(t)}),
$$
with population $P^{(t)}=\{h_1^{(t)},\dots,h_M^{(t)}\}$ and control vector $\theta^{(t)}$. Offspring generation is modeled by
$$
h_c = \mathcal{L}(\Pi(\rho(h_p), \theta^{(t)})), \qquad O^{(t)}=\{h_{c,1},\dots,h_{c,N}\},
$$
where $\mathcal{L}$ is the LLM, $\Pi$ is prompt construction, and $\rho(h_p)$ is the symbolic representation of parent heuristics. Selection then produces the next population through
$$
P^{(t+1)} = \mathcal{S}(U^{(t)}; g),
$$
with $U^{(t)} = P^{(t)} \cup O^{(t)}$ [2508.13333].

Within this formulation, the search objects are concrete heuristic components. In the reported tasks, these include a next-city decision rule for step-by-step TSP construction, a penalty-update heuristic for TSP with guided local search, a bin-assignment scoring function for online bin packing, a perturbation/guidance function for flow shop scheduling after local optima, and a cost-aware acquisition or utility function for Bayesian optimization [2508.13333]. The method therefore searches over **algorithmic code and strategy**, not over fixed parameter vectors alone.

## 3. Dual prompting architecture

HiFo-Prompt’s defining contribution is the coupling of a **Foresight module** with a **Hindsight module**. The first controls what the search should emphasize next; the second preserves what the search has already learned.

The Foresight module is implemented as the **Evolutionary Navigator**. It monitors three population-level signals: consecutive stagnation count $C_{\text{stag}}(t)$, consecutive progress count $C_{\text{prog}}(t)$, and population diversity $\Delta_p(t)$, defined as the standard deviation of fitness values. The state-dependent strategy is
$$
\theta_t =
\begin{cases}
\theta_{\text{explore}} & \text{if } C_{\text{stag}}(t)\geq T_{\text{stag}} \text{ or } \Delta_p(t)<\Delta_{\text{crit}} \\
\theta_{\text{exploit}} & \text{if } C_{\text{prog}}(t)\geq T_{\text{prog}} \\
\theta_{\text{balance}} & \text{otherwise.}
\end{cases}
$$
In the reported experiments, $T_{\text{stag}}=3$, $T_{\text{prog}}=2$, and $\Delta_{\text{crit}}=0.3$ [2508.13333]. The resulting regime is translated into a **Design Directive** inserted into the next-generation prompt. Explore directives emphasize novelty and alternative methodologies; exploit directives emphasize refinement of existing high-performing heuristics; balance directives combine both tendencies.

The Hindsight module is the **Insight Pool**, a bounded memory of distilled design principles extracted from elite heuristics. At the end of a generation, the system prompts an LLM to summarize the top 30% of the population into **1–2 concise, generic, performance-positive design principles**. These are admitted only if their Jaccard similarity to existing pool entries is below a novelty threshold of $0.7$. Each stored insight carries an effectiveness score, usage statistics, and recency information. Retrieval uses an adaptive utility that combines effectiveness, a usage penalty, and a recency bonus; in experiments, the pool capacity is $30$, retrieval count is $s=3$, usage-penalty weight is $0.1$, and recency-bonus parameter is $0.2$ [2508.13333].

The Hindsight module is not merely archival. When an offspring was generated using retrieved insights, those insights receive credit based on the offspring’s population-relative performance, and their effectiveness is updated by exponential moving average with $\alpha=0.3$. Eviction applies a decay term with $R_{\text{decay}}=0.01$, while newly added insights are protected for a grace period of $3$ generations [2508.13333]. This turns transient discoveries into a persistent, scored, reusable knowledge base.

## 4. Evolutionary workflow and guided prompt synthesis

The operational pipeline begins with either seed heuristics or LLM-generated initial heuristics. The system then initializes the Insight Pool and the Evolutionary Navigator, and runs an evolutionary loop over a small population of size $4$ [2508.13333]. The foundational operator prompts are fixed and correspond to six heuristic-generation modes:

- **I1**: initialize from scratch  
- **E1**: recombination into a structurally novel algorithm  
- **E2**: recombination via a shared backbone  
- **M1**: structural mutation  
- **M2**: parameter mutation  
- **M3**: simplification mutation

For each active operator, the system performs what the paper calls **Guided Prompt Synthesis**. The final generation prompt is a composite of four elements: the foundational operator prompt, parent heuristic representations, retrieved hindsight insights, and the current foresight design directive [2508.13333]. In effect, the LLM is asked not only to modify or combine heuristics, but to do so under explicit strategic regulation and with access to distilled historical knowledge.

Parent selection is rank-based, with probabilities proportional to $1/(k+1+N)$ after sorting the population by fitness. The generated offspring are evaluated, invalid or duplicate individuals are filtered, and survival selection retains the top $N_{\text{pop}}$ heuristics [2508.13333]. The reported implementation uses **qwen2.5-max** through **DashScope**, with temperature $1.0$, and runs for **8 generations** on the combinatorial optimization tasks and **4 generations** on Bayesian optimization [2508.13333].

A notable design choice is the decoupling of **thought** from **code**. The framework extracts and stores high-level design principles independently of the exact implementation that first expressed them. This suggests a move away from memoryless prompt-and-select loops toward a genuinely experience-accumulating search process [2508.13333].

## 5. Empirical performance and ablation evidence

The evaluation covers TSP, online bin packing, flow shop scheduling, and Bayesian optimization, with comparisons against handcrafted heuristics, neural combinatorial optimization baselines, and prior LLM-based automatic heuristic design systems [2508.13333]. The reported pattern is consistent: HiFo-Prompt outperforms prior LLM-based AHD baselines across the main settings, and often by large margins.

Representative results illustrate the scale of the gains:

| Setting | Strong prior result | HiFo-Prompt |
|---|---:|---:|
| TSP20 step-by-step | ReEvo 6.156% | **2.790%** |
| TSP100 GLS | EoH 0.026% | **0.014%** |
| Online BPP, capacity 300, 5k | MCTS-AHD 0.95% | **0.63%** |
| FSSP, \(n=100,m=20\) | EOH 0.927% | **0.472%** |
| TSPLib OOD mean gap | ReEvo 17.401% | **12.843%** |

In step-by-step TSP construction, the gap at TSP20 falls from $9.236\%$ for EoH and $6.156\%$ for ReEvo to $2.

Source: https://www.emergentmind.com/topics/hifo-prompt