We Used RL, but … Did It Work?!

• Activity Number: 2 - When Causal Inference Meets Reinforcement Learning: The Story of Mobile-Delivered Interventions
• Type: Invited
• Date/Time: Sunday, August 8, 2021 : 1:30 PM to 3:20 PM
• Sponsor: IMS
• Author(s): Peng Liao* and Susan Murphy and Predrag Klasnja
• Companies: Harvard University and Harvard University and University of Michigan
• Keywords: reinforcement learning; mobile health; online learning

Abstract

Recent progress in mobile health technology allows health scientists to collect real-time sensor data and deliver sequences of treatment anytime and anywhere. The right timing and type of treatment ideally should adapt to the user’s changing state to maximize the efficacy of the treatments. Reinforcement Learning (RL) is a natural method for continuously learning, as the user experiences the treatments, when and in which state it is best to deliver treatments. Once the RL algorithm is run on multiple users, an important question to address is how to gather evidence to obtain first indications of whether the RL worked, that is, personalized to the user. In this work, we provide two types of methods to address this question. The first method borrows ideas from meta-analysis and the second method is based on resampling. This work is motivated and illustrated by HeartSteps, a physical activity mobile health intervention.