Herakles in Astrophysics, AI, and Galactic Dynamics

• Herakles is a multi-domain framework that spans computational astrophysics, AI hierarchical skill compilation, and galactic dynamics by addressing distinct research challenges.
• In astrophysics, the HERAKLES code simulates complex stellar convection and nuclear burning using PPM hydrodynamics and operator splitting, achieving close alignment with theoretical convective velocities and entrainment dynamics.
• In AI, the HERAKLES HRL framework employs a two-level system that compiles high-level language skills into efficient low-level policies, significantly reducing sample complexity and improving goal mastery.

Herakles refers to a set of highly distinct concepts spanning computational astrophysics, artificial intelligence, and galactic dynamics—each with active research representation in the academic literature. These include (1) a hydrodynamics code for stellar convection and nuclear burning in astrophysical contexts (Mocak et al., 2010, Mocak et al., 2011), (2) a hierarchical skill compilation framework for open-ended LLM agents (Carta et al., 20 Aug 2025), and (3) the identification of “trojan” orbits in galactic bar dynamics underlying the Hercules stellar stream (D'Onghia et al., 2019). Each instance of Herakles is foundational within its domain, demanding a precise technical overview.

1. HERAKLES in Computational Astrophysics: Hydrodynamics of Stellar Interiors

Herakles is a parallel, modular, finite-volume hydrodynamics code engineered to model convection and nuclear burning in stellar interiors, particularly within regimes where one-dimensional (1D) stellar evolution is inadequate (Mocak et al., 2010, Mocak et al., 2011). The code solves the fully compressible reactive Euler equations with self-gravity and volumetric nuclear source terms in spherical coordinates $(r, \theta, \phi)$, coupling mass, momentum, energy, and species conservation equations:

• $\partial_t \rho + \nabla \cdot (\rho \vec{v}) = 0$
• $$\partial_t (\rho \vec{v}) + \nabla \cdot (\rho \vec{v} \otimes \vec{v} + PI) = -\rho \nabla \Phi$$
• $$\partial_t E + \nabla \cdot [(E+P)\vec{v}] = -\rho \vec{v}\cdot\nabla \Phi + \rho \dot{\epsilon}_{\text{nuc}}$$
• $$\partial_t (\rho X_i) + \nabla \cdot (\rho X_i \vec{v}) = \rho (\dot{M}_i)_{\text{nuc}}$$

The piecewise parabolic method (PPM) of Colella and Woodward is employed for reconstruction, with an approximate real-gas Riemann solver at interfaces. Nuclear source terms and self-gravity are incorporated via operator splitting, using a semi-implicit Bader–Deuflhard integrator for the nuclear network.

Herakles operates in wedge geometries to maximize spatial resolution across convective shells, with typical resolutions in 3D simulations for the helium flash of $270 \times 30 \times 30$ zones (Pop I stars) or $720\times 50\times 50$ for Pop III stars. Boundary conditions are reflecting in the radial direction and transmissive or periodic in angle.

Code parallelization is realized via MPI-based domain decomposition, supporting per-wedge load balancing. Validation studies demonstrate robust energy conservation and order-of-magnitude agreement with mixing length theory for convective velocities, albeit with systematically lower Reynolds numbers and some numerical viscosity (Mocak et al., 2010).

2. Applications to Core Helium Flash and Shell Convection

Herakles has furnished critical insights into the multi-dimensional dynamics of the core helium flash in both metal-rich Pop I and metal-free Pop III low-mass stars (Mocak et al., 2010). Key scientific findings include:

• In Pop I models, a single large convection zone emerges, expanding dynamically through turbulent entrainment at both inner and outer boundaries. The predicted hydrogen injection episode commences in $\sim$17–23 days, likely precipitating the development of a double convection zone.
• For Pop III models, double convection zones—inner helium-driven and outer hydrogen-CNO–driven—are initially established but decay rapidly ($\lesssim 10^3$–$10^4$ s), ceding to internal gravity wave dominance.
• Convection zones in 3D display plume-like velocity structures, while 2D simulations exhibit larger-scale, overestimated velocities, reflecting dimensionality effects.
• A newly discovered dynamic mixing process, characterized by the descent of narrow, cold, high–mean molecular-weight blobs in regions with $\partial_t \rho + \nabla \cdot (\rho \vec{v}) = 0$0 and marginally stable stratification, was observed, with possible origins in $\partial_t \rho + \nabla \cdot (\rho \vec{v}) = 0$1-gradient instabilities or shear-induced turbulence (Mocak et al., 2011).

Herakles’s physics fidelity—resolving convective boundary entrainment, distinguishing gravity-wave phenomena, and incorporating detailed nuclear reaction networks—renders it a reference tool for flash-driven convection studies.

3. HERAKLES: Hierarchical Skill Compilation in Open-ended LLM Agents

HERAKLES (HiERarchicAl sKill compiLation for open‐Ended LLM agentS) denotes a two-level hierarchical reinforcement learning (HRL) framework in open-ended AI (Carta et al., 20 Aug 2025). HERAKLES enables autotelic agents to acquire and efficiently reuse skills over a growing and diversifying goal space, circumventing the sample complexity explosion inherent to flat, goal-conditioned RL approaches.

System Structure

• High-level controller ($\partial_t \rho + \nabla \cdot (\rho \vec{v}) = 0$2): A LLM (LLM, e.g., Mistral 7B fine-tuned with LoRA) that, at each decision point, selects among available subgoals/skills based on observation and target goal, executing constrained decoding over an admissible set $\partial_t \rho + \nabla \cdot (\rho \vec{v}) = 0$3.
• Low-level executor ($\partial_t \rho + \nabla \cdot (\rho \vec{v}) = 0$4): A compact neural policy trained with advantage-weighted regression (AWR), capable of executing either primitive actions or compiled skills.

Whenever a composite goal $\partial_t \rho + \nabla \cdot (\rho \vec{v}) = 0$5 is mastered, the entire primitive trajectory is distilled (“compiled”) into the low-level policy as a single reusable skill, expanding the admissible high-level action space.

The agent operates in a goal-conditioned POMDP $\partial_t \rho + \nabla \cdot (\rho \vec{v}) = 0$6 with RL objectives defined at both levels. Competence in candidate skills is estimated by a learned, LLM-based success predictor. Option selection is filtered: only mastered or improving skills are exposed to high-level selection, maintaining sample efficiency as the complexity of the compositional goal tree grows.

Empirical Performance

HERAKLES was benchmarked on the Crafter environment (a 2D Minecraft-like framework) against strong HRL baselines (Carta et al., 20 Aug 2025). Key findings:

• The number of high-level actions required to master goals grows quasi-linearly with task difficulty for HERAKLES, in contrast to exponential escalation for flat or embedding-based HRL.
• HERAKLES achieves mastery of $\partial_t \rho + \nabla \cdot (\rho \vec{v}) = 0$78 out of 10 goals in 30k high-level steps, with robust generalization to semantically variant or compositional tasks, outperforming analogues by 20–80 p.p.
• Ablations show critical dependence on high-level updates, language exposure, and competence filtering.

Limitations include the necessity for retraining low-level policy on new skills, focus on textual (not visual) goals, and the assumption of static goal spaces.

4. Herakles and the Hercules Stream: Galactic Bar “Trojans”

In galactic dynamics, Herakles is closely associated with the dynamics of the Hercules stellar stream, interpreted as an assembly of “trojan” stars captured at the L4 Lagrange point of the Galactic bar (D'Onghia et al., 2019). N-body simulations (GALAKOS) demonstrate that:

• Hercules stream stars are predominantly on orbits in near co-rotation resonance with the bar, specifically, those “librating” around the L4 point in the bar’s rotating frame.
• The phase-space structure is set by the effective potential $\partial_t \rho + \nabla \cdot (\rho \vec{v}) = 0$8 and corresponding Hamiltonian.
• Capture into the trojan branch is transient; typical libration lifetimes are $\partial_t \rho + \nabla \cdot (\rho \vec{v}) = 0$9500 Myr, after which stars escape L4 due to bar slowdown and resonance evolution.
• Retrograde libration (the so-called “donkey” orbits) produces a positive skew in the radial velocity distribution at azimuths near the Sun, in quantitative agreement with kinematic signatures observed in Gaia data.

Predictions include spatial depletion of Hercules along the bar’s major axis and enhancement “behind” the Sun at azimuths toward L4, along with the anti-Hercules stream at L5—features distinguishing the trojan scenario from outer Lindblad resonance (OLR) models (D'Onghia et al., 2019).

5. Technical and Methodological Summary

Instance	Domain	Core Features/Method
HERAKLES code	Astrophysics (convection, flash)	PPM hydro, stiff nuclear coupling, MPI parallel
HERAKLES HRL	AI, HRL/LLM agents	LLM high-level, skill compilation, RL
Hercules/Trojans	Galactic dynamics	N-body, resonance trapping, spectral analysis

The three Herakles implementations are unified by their technical sophistication, multi-scale hierarchical structure (explicitly so in both code and HRL agent), and prominence in their respective research areas.

6. Open Problems and Future Directions

For the HERAKLES hydrodynamics code, critical open issues include incorporation of rotation, magnetic fields, and more complete nuclear networks, as well as improved subgrid-scale turbulence modeling and real-time gravity coupling (Mocak et al., 2011). For HERAKLES HRL agents, future extensions involve continual skill consolidation, integration with autonomous goal generators, and expansion to vision-language grounded environments (Carta et al., 20 Aug 2025). In galactic dynamics, further spectroscopic and kinematic mapping of Hercules, and validation against forthcoming Gaia data releases, will test the trojan origin versus alternative resonance interpretations (D'Onghia et al., 2019).

Continued research on and with Herakles in each manifestation promises substantive advances at the physics–computation, cognition–action, and dynamics–structure interfaces.