An Actor-Critic Contextual Bandit Algorithm for Personalized Interventions Using Mobile Devices

• Activity Number: 41
• Type: Contributed
• Date/Time: Sunday, August 9, 2015 : 2:00 PM to 3:50 PM
• Sponsor: Section on Statistical Learning and Data Mining
• Author(s): Huitian Lei* and Ambuj Tewari and Susan A. Murphy
• Companies: University of Michigan and University of Michigan and University of Michigan
• Keywords: Contextual bandit ; Personalized treatment ; Mobile health

Abstract

A Just-In-Time Adaptive Intervention (JITAI) employs the real-time data collection and communication capabilities that modern mobile devices provide to adapt and deliver interventions in real-time. The lack of methodological guidance in constructing data-based high quality JITAI remains a hurdle in advancing JITAI research despite the increasing popularity JITAIs receive from clinical and behavioral scientists. We make a first attempt to bridge this methodological gap by formulating the task of tailoring interventions in real-time as a contextual bandit problem. We choose the reward function (the "critic") parameterization separately from a lower dimensional parameterization of stochastic policies (the "actor"). We provide an online actor-critic algorithm that guides the construction and refinement of a JITAI. Statistical properties of actor-critic algorithm, including consistency, asymptotic distributions and confidence intervals of JITAI parameters are provided and verified by numerical experiments. To the best of our knowledge, our algorithm is the first application of the actor-critic architecture to contextual bandit problems.