Helix: Evolutionary Reinforcement Learning for Open-Ended Scientific Problem Solving

Abstract: LLMs with reasoning abilities have demonstrated growing promise for tackling complex scientific problems. Yet such tasks are inherently domain-specific, unbounded and open-ended, demanding exploration across vast and flexible solution spaces. Existing approaches, whether purely learning-based or reliant on carefully designed workflows, often suffer from limited exploration efficiency and poor generalization. To overcome these challenges, we present HELIX -- a Hierarchical Evolutionary reinforcement Learning framework with In-context eXperiences. HELIX introduces two key novelties: (i) a diverse yet high-quality pool of candidate solutions that broadens exploration through in-context learning, and (ii) reinforcement learning for iterative policy refinement that progressively elevates solution quality. This synergy enables the discovery of more advanced solutions. On the circle packing task, HELIX achieves state-of-the-art result with a sum of radii of 2.63598308 using only a 14B model. Across standard machine learning benchmarks, HELIX further surpasses GPT-4o with a carefully engineered pipeline, delivering an average F1 improvement of 5.95 points on the Adult and Bank Marketing datasets.

• The paper introduces Helix, a closed-loop system that combines GRPO policy updates, NSGA-II selection for reward and diversity, and lineage-aware prompts to improve scientific program search.
• Helix achieves the best results on 17 of 20 tasks, including a reported 2.63598308 sum of radii for 26-circle packing and major gains on machine-learning and physics benchmarks.
• Ablation studies show that both policy learning and diversity-preserving evolutionary memory are necessary, while limitations include high compute costs, reliance on verifiable rewards, and evaluation on one model family.

Motivation and problem setting

The paper addresses the application of LLMs to complex scientific problems, which the authors characterize along three axes: they are domain-specific (each task has unique constraints and environments), open-ended (solution spaces are vast and flexible), and unbounded (no known or guaranteed global optimum). The authors argue that an effective LLM for such tasks must support three capabilities: learning from experience via task-specific policy adaptation, balancing solution quality and diversity during exploration, and iteratively building upon previously discovered high-quality solutions.

They position existing approaches as falling short on these criteria. Post-training methods such as SFT and RLVR suffer from entropy collapse and, citing prior evidence that RL rarely extends beyond base-model capabilities, rarely discover fundamentally new solutions—particularly under sparse rewards. Workflow-driven evolutionary systems (e.g., AlphaEvolve-style pipelines, LLaMEA, GEPA) are effective on narrow tasks but are sensitive to workflow design and cannot reuse past discoveries to guide iterative search because the underlying policy remains static. Helix is proposed as a hybrid that unifies reinforcement learning and evolutionary search within a single closed loop.

Method

Helix treats each scientific task as code-space optimization: a candidate solution $s \in \mathcal{S}$ is a program in some DSL (Python, YAML, etc.), evaluated by a verifiable reward function $R(\cdot)$ depending only on the current solution. The framework has three coupled components:

1. RL-based policy refinement. The LLM policy $\pi_\theta$ acts as a parameterized mutation operator, conditioned on a prompt containing the problem description, the current solution, its reward, evaluator feedback, and up to $n$ ancestral solutions along its lineage tree. Policy updates use GRPO with clipped token-level ratios, group-normalized advantages over $G$ rollouts, and a KL penalty against a reference policy. Because rollouts from GRPO naturally form a population, policy optimization and evolutionary search share data in a closed loop.
2. Multi-objective evolutionary selection. All generated solutions accumulate into a dataset $\mathcal{D}$; the candidate pool $\mathcal{P}_t$ is selected by NSGA-II non-dominated sorting over two objectives: reward and diversity. Diversity is computed as one minus the average cosine similarity between a solution's embedding (from a pretrained embedding model after canonicalization) and its $k$ nearest neighbors in embedding space. This yields a Pareto front of high-reward yet semantically distinct candidates, mitigating entropy collapse and mode collapse in the population.
3. In-context experiences. Prompts embed rewards and structured feedback of ancestors, enabling the model to build on prior discoveries rather than re-explore from scratch.

Training uses DeepSeek-R1-Distill-Qwen backbones (14B for most tasks, 32B for physics simulation), fine-tuned with VERL/GRPO for 80 epochs at learning rate $10^{-6}$, KL coefficient $10^{-3}$, 16 rollouts per group, on 8 A100s (14B) or 16 H100s (32B).

Empirical results

Evaluation spans 20 tasks across five categories: machine learning (Adult Income, Bank Marketing, Boston Housing, Transparent Conductors), physics simulation via COMSOL (inductor design, beam bending, magnetic torque, periodic heat, acoustic demultiplexer), circle packing (26 circles in unit square/disk), function minimization (Eggholder, Mishra's Bird, Keane's Bump in 10/20/30 dimensions), and symbolic regression on LLM-SRBench. Baselines include direct prompting with best-of-64 sampling, OpenEvolve (an open-source AlphaEvolve reproduction), GPT-4o with engineered pipelines, and human-crafted task-specific methods (LightGBM/RRL, COMSOL parameter scan/topology optimization, SLSQP/GA, SLSQP/trust-constr, LLM-SR/LaSR).

Headline claims:

• Helix achieves the best result on 17 of 20 tasks, outperforming all baselines; it beats OpenEvolve on 19 tasks and GPT-4o on 18 tasks.
• On circle packing in a unit square ($R(\cdot)$0), Helix reports a sum of radii of 2.63598308, claimed as a new world record, using only a 14B model.
• On Transparent Conductors (a Kaggle competition dataset), Helix attains RMSLE 0.049, reported as second-highest on the participants' leaderboard.
• On Boston Housing, Helix reaches RMSE 1.747 versus 2.937–3.258 for all baselines—a substantial margin.
• On physics tasks where weak base models nearly fail (e.g., Qwen direct prompting scores 0.323 on magnetic torque vs. 11.045 for Helix; Bank Marketing F1 improves from 0.00 to 80.65), parameter updates plus in-context learning recover strong performance, indicating the framework can partially compensate for weaker base models.
• Convergence analysis shows both average reward and output validity rising monotonically during training on representative tasks.

Ablations isolate component contributions. Variants retaining only top-score selection (TopScore), only diversity (TopDiv), random selection, evolution-only (frozen weights), or training-only (pure GRPO without evolution/in-context prompting) all underperform the full system. On Circle Packing, removing diversity causes collapse into narrow modes, while diversity-only selection introduces instability; on Boston Housing, EvoOnly plateaus at initial model capacity and TrainOnly collapses, establishing that both weight updates and in-context evolutionary guidance are necessary. Scaling experiments across 1.5B–32B models show reward increasing with scale on magnetic torque, while inductor design exhibits a max-reward plateau near 9.6 but continued growth in mean reward—suggesting larger models produce more valid candidates even when peak performance saturates.

Analysis of why naive RL–EA integration fails

An appendix case study contrasts Helix against the sequential AlphaEvolve paradigm on Circle Packing. OpenEvolve with Qwen-14B scored 1.586—below the 1.673 direct-prompting baseline despite far more compute. Two failure modes are identified: initialization bias, where few seed solutions trap evolution in local optima, and rejection of novelty requiring destructive changes—in a 4,147-trial run, only 0.3% of trials attempted a scipy-based optimization approach, and all 12 were discarded due to transient compilation errors despite the approach's potential to exceed 2.0 once debugged. Helix's embedding-based diversity scoring preserves such low-reward-but-novel candidates, and RL then consolidates successful realizations into policy parameters, forming a positive feedback loop.

Theoretical analysis

The paper provides a simplified theoretical treatment. Modeling LLM transitions as Gaussian perturbations in embedding space and the reward landscape as two Gaussian peaks, a theorem shows that memory-less GRPO converges to a local optimum whenever optima are sufficiently separated ($R(\cdot)$1), the local peak is not too weak, and initialization lies closer to the local basin. Further, modeling Evolve as selection-diffusion and Helix as drift-diffusion, stationary distribution analysis yields concentration scaling of $R(\cdot)$2 for Evolve versus $R(\cdot)$3 for Helix under noise level $R(\cdot)$4, implying exponentially tighter concentration around optima for Helix as $R(\cdot)$5. These results rest on strong idealizations—an injective, continuous, open-map embedding, Gaussian transition approximations, and bounded solution spaces—and should be read as qualitative justification rather than tight predictions for real code spaces.

Limitations and open questions

Several limitations are acknowledged or evident. The evaluation relies on a single model family (DeepSeek-R1-Distill-Qwen); generalization to other backbones is not established. Physics tasks require the 32B variant, indicating that geometric reasoning demands may exceed smaller models regardless of the framework. The inductor scaling plateau suggests diminishing returns of model size on some tasks, and the mechanism behind this saturation is not analyzed. The theoretical results depend on assumptions (Gaussian transitions, well-behaved embeddings) whose validity for realistic code distributions is unverified. Compute costs of full-parameter RL fine-tuning relative to inference-only evolutionary baselines are not systematically quantified. Finally, whether the diversity metric—kNN distance in a generic embedding space—remains informative as populations grow very large is left open, as is the question of how the framework behaves when rewards are noisy or expensive to evaluate rather than fast and deterministic.

Conclusion

Helix integrates GRPO-based policy learning, NSGA-II multi-objective selection over reward and embedding-based diversity, and lineage-aware in-context prompting into a unified loop for open-ended scientific optimization. Across 20 heterogeneous tasks it outperforms direct prompting, OpenEvolve-style evolution, GPT-4o pipelines, and human-designed task-specific methods on the majority of benchmarks, including a reported record of 2.63598308 on 26-circle packing in a unit square with a 14B model. The central empirical lesson—that explicit diversity accounting preserves nascent innovations while RL consolidates them into parameters—addresses concrete failure modes of naive evolutionary pipelines, though the framework's dependence on verifiable rewards, specific model families, and substantial training compute bounds its current scope.