Selection Rules for Species Coexistence in a Hierarchical May-Leonard Model

Abstract: One of the central challenges in evolutionary dynamics is understanding why some species combinations persist while others disappear. Although cyclic-interaction models have provided fundamental insights into biodiversity maintenance, much less is known about how hierarchical competitive interactions shape long-term community organization. Here, we investigate a hierarchical extension of the May-Leonard model, in which species interact through a directed predation chain while undergoing reproduction and mortality. Combining mean-field analysis with Monte Carlo simulations, we show that the fully coexisting state is generically unstable, causing the dynamics to evolve toward lower-dimensional coexistence states. The simulations further reveal stochastic extinctions dominating small populations with the dynamics progressively approaching the mean-field predictions as the system size increases. Rather than permitting arbitrary species combinations, the hierarchical-interaction structure dynamically constrains coexistence by selecting only specific subsets of species for long-term persistence. We show that these admissible coexistence states have a natural graph-theoretic interpretation as independent sets in the hierarchical interaction network, thereby providing general constraints on coexistence in hierarchical communities. Together, these results establish a theoretical framework linking hierarchical interactions, dynamical selection, graph topology, and biodiversity organization, extending the classical May-Leonard model beyond cyclic competition.