Optimal control of stochastic reaction networks with entropic control cost and emergence of mode-switching strategies (2409.17488v1)

Abstract: Controlling the stochastic dynamics of biological populations is a challenge that arises across various biological contexts. However, these dynamics are inherently nonlinear and involve a discrete state space, i.e., the number of molecules, cells, or organisms. Additionally, the possibility of extinction has a significant impact on both the dynamics and control strategies, particularly when the population size is small. These factors hamper the direct application of conventional control theories to biological systems. To address these challenges, we formulate the optimal control problem for stochastic population dynamics by utilizing a control cost function based on the Kullback-Leibler divergence. This approach naturally accounts for population-specific factors and simplifies the complex nonlinear Hamilton-Jacobi-BeLLMan equation into a linear form, facilitating efficient computation of optimal solutions. We demonstrate the effectiveness of our approach by applying it to the control of interacting random walkers, Moran processes, and SIR models, and observe the mode-switching phenomena in the control strategies. Our approach provides new opportunities for applying control theory to a wide range of biological problems.

• The paper introduces a KL divergence-based control scheme that linearizes the Hamilton–Jacobi–Bellman equation for managing stochastic reaction networks.
• It demonstrates analytic solutions for interacting random walkers, Moran processes, and stochastic SIR models to direct complex biological dynamics.
• The research reveals mode-switching strategies that adjust control intensity, optimizing intervention under varying population states.

Optimal Control of Stochastic Reaction Networks with Entropic Control Cost and Emergence of Mode-Switching Strategies

The paper "Optimal control of stochastic reaction networks with entropic control cost and emergence of mode-switching strategies" by Shuhei A. Horiguchi and Tetsuya J. Kobayashi addresses the complex problem of orchestrating stochastic population dynamics. By leveraging a control cost function rooted in the Kullback–Leibler (KL) divergence, the authors propose an approach that simplifies the inherently nonlinear Hamilton–Jacobi–BeLLMan (HJB) equation into a more tractable linear form. This methodological advance allows for effective and efficient computation of optimal control strategies across various biological systems, ranging from interacting random walkers to epidemic models.

Theoretical Framework and Methodology

The central objective of the paper is to formulate and solve the optimal control problem for stochastic reaction networks (RNs), which are characterized by their intrinsic nonlinearity, discrete state space, and potential for extinction events. Conventional control theories primarily tailored to diffusion processes fall short in addressing these challenges. This research overcomes that gap by devising an optimal control scheme built around the KL divergence. The choice of KL divergence as a control cost function is particularly advantageous as it naturally accommodates the non-negativity constraints of control parameters and state variables, while also enabling the linearization of the nonlinear HJB equation through the Cole-Hopf transformation.

Application to Biological Systems

The validity and practicality of the proposed framework are demonstrated through three distinct biological phenomena:

1. Interacting Random Walkers: The first example involves the control of one-dimensional stochastic random walks. The authors derive analytic solutions for scenarios where particles are required to reach a specific goal while accounting for exclusion interactions. This problem, relevant to intracellular transport, exemplifies how optimal control in RNs can efficiently direct molecular motors along cellular filaments.
2. Moran Processes: The second example involves the control of Moran processes representing populations of two competing species. By utilizing the analytical properties of KL control cost, the authors identify mode-switching phenomena in optimal control strategies. When the population exhibits exponential growth, hierarchical control strategies with alternating ON and OFF modes emerge, reflecting a nuanced balancing act between maintaining diversity and controlling population size.
3. Stochastic SIR Models: The third example applies the method to pandemic control in a stochastic SIR (Susceptible-Infected-Recovered) model. The research illustrates that under high $R_0$ values, the optimal control strategy involves strong initial intervention to reduce transmission, followed by a transition to minimal control when the infection has already spread extensively. This mode-switching dynamic adapts the level of control according to the current state of the epidemic, optimizing the allocation of limited resources.

Numerical Results and Analysis

In addition to deriving theoretical results, the paper presents comprehensive numerical analyses. These include value functions, control cost trade-offs, and gradient fields, which collectively elucidate the nature of the optimal control policies. Notably, the mode-switching behaviors are highlighted in the control of Moran processes and epidemic models, underscoring the flexibility of the proposed method in managing biologically relevant dynamics.

Implications and Future Directions

The proposed framework opens significant avenues for applying control theory to a broad spectrum of biological problems involving stochastic dynamics. Practically, this can influence fields like cancer therapy, stem cell culturing, and epidemic control, where managing population dynamics under uncertainty is crucial. Theoretically, the approach sets a precedent for embedding entropic measures into control cost functions, promoting more realistic and computationally feasible control strategies.

Future research could explore risk-sensitive extensions of the current model, aiming to incorporate variability and uncertainties more robustly. Additionally, addressing scalability issues for large and high-dimensional systems through advanced numerical techniques, such as Monte Carlo sampling, remains an important frontier. Customizing the control cost function to fit more diverse and specific biological settings while maintaining tractability is another intriguing possibility.

In conclusion, this paper presents a substantial advancement in the field of optimal control theory for stochastic reaction networks. By employing a KL divergence-based control cost, the authors effectively bridge the gap between theoretical optimal control and practical applicability in complex biological systems.