MiWaves Reinforcement Learning Algorithm (2408.15076v1)

Abstract: The escalating prevalence of cannabis use poses a significant public health challenge globally. In the U.S., cannabis use is more prevalent among emerging adults (EAs) (ages 18-25) than any other age group, with legalization in the multiple states contributing to a public perception that cannabis is less risky than in prior decades. To address this growing concern, we developed MiWaves, a reinforcement learning (RL) algorithm designed to optimize the delivery of personalized intervention prompts to reduce cannabis use among EAs. MiWaves leverages domain expertise and prior data to tailor the likelihood of delivery of intervention messages. This paper presents a comprehensive overview of the algorithm's design, including key decisions and experimental outcomes. The finalized MiWaves RL algorithm was deployed in a clinical trial from March to May 2024.

• The paper introduces the MiWaves RL algorithm that personalizes intervention prompts to reduce cannabis use among emerging adults.
• It employs a Bayesian Mixed Linear Model with strategic daily and weekly updates to optimize message delivery based on user engagement and context.
• Clinical trial evaluations reveal superior performance in capturing participant heterogeneity compared to fully pooled models, advancing mHealth intervention strategies.

Overview of MiWaves Reinforcement Learning Algorithm

The MiWaves Reinforcement Learning (RL) algorithm represents a novel approach to optimizing personalized intervention prompts aimed at reducing cannabis use among emerging adults (EAs) aged 18-25. Designed by Susobhan Ghosh et al., MiWaves leverages prior data and domain expertise to effectively tailor intervention message delivery, enhancing user engagement with the intervention app. This essay provides an in-depth examination of the MiWaves algorithm, focusing on its design, theoretical underpinnings, empirical evaluation, and implications for future AI developments in mobile health (mHealth) interventions.

Introduction

The motivation behind MiWaves is rooted in the increasing prevalence of cannabis use among EAs in the United States, a demographic particularly susceptible to early and more intense substance use following cannabis legalization in multiple states. Given the public health implications, reducing cannabis use in this group is paramount. MiWaves addresses this concern through an RL algorithm that optimizes intervention message delivery based on user-specific contexts.

Design and Formulation

The core of MiWaves is its RL algorithm, which formulates the problem through states, actions, and rewards:

• States ($\state{t}{i}$): Characterize the participant's context, incorporating features such as recent engagement with the app, the time of day, and recent cannabis use.
• Actions ($\action{t}{i}$): Binary decisions to send an intervention prompt ($\action{t}{i}=1$) or not ($\action{t}{i}=0$).
• Rewards ($\reward{t+1}{i}$): Functions of proximal outcomes, primarily app engagement metrics such as check-in completions.

MiWaves employs a Bayesian Mixed Linear Model to approximate the reward, which adapts to individual participants while incorporating population-wide trends. The reward model is updated periodically through a detailed posterior update mechanism, ensuring that it reflects the latest participant interactions.

Experimental Design

The MiWaves algorithm was tested in a clinical trial over 30 days involving 122 participants, with decisions made twice daily. The RL algorithm's performance was evaluated using several metrics:

• Average total reward per participant.
• Median total reward.
• Average and median total reward for the lower 25th percentile of participants.

Results and Key Findings

The empirical evaluation used multiple simulation environments inspired by a similar dataset from the Substance Use Research Assistant (SARA) paper. Various algorithm variants were tested, differing in their baseline and advantage functions, the presence of random effects, and the update cadence for hyper-parameters and posteriors. Notably:

• Algorithms with mixed effects models outperformed fully pooled models, indicating the importance of capturing participant heterogeneity.
• Baseline and advantage functions incorporating all interactions (Variant 0) performed comparably well, suggesting that more complex models do not necessarily incur performance penalties.
• Daily updates for posteriors and weekly updates for hyper-parameters struck an optimal balance between computational efficiency and algorithm performance.

Implications and Future Directions

MiWaves illustrates the potential of RL in delivering personalized mHealth interventions. Its design allows for continuous learning and adaptation, making it particularly suited for dynamic health interventions. The approach of using mixed models to incorporate both individual-specific and population-wide trends can be generalized to other domains within AI and mHealth.

Future developments could explore:

• Integrating more sophisticated state representations, such as those derived from wearable sensor data.
• Extending the algorithm to other types of substance use interventions.
• Investigating multi-objective RL frameworks to balance engagement with actual reductions in substance use.

Conclusion

The MiWaves RL algorithm presents a methodologically rigorous approach to optimizing personalized interventions for cannabis use reduction. Its successful deployment in a clinical trial underscores its practical applicability and robustness. This work sets a precedent for future AI-based health interventions, showcasing how domain expertise and advanced machine learning techniques can converge to address critical public health challenges.