Parameterizing Exploration

• Activity Number: 112 - Statistical Challenges in the Processing and Analysis of Mobile Health Data
• Type: Invited
• Date/Time: Monday, July 29, 2019 : 8:30 AM to 10:20 AM
• Sponsor: Section on Statistics in Epidemiology
• Author(s): Jesse Clifton* and Lili Wu and Eric B Laber
• Companies: NC State University and North Carolina State University and NC State University
• Keywords: reinforcement learning; mobile health; machine learning; multi armed bandit

Abstract

Exploration in reinforcement learning is often accomplished by heuristics which account for neither the time horizon of the decision problem nor the decision-maker’s current state of knowledge of the dynamics of the underlying system. Accounting for these features could greatly improve the exploration-exploitation tradeoff (for instance, by avoiding over-exploration in short time horizons), and is computationally feasible in applications such as mobile health (mHealth) in which moderate computing times are acceptable. We introduce Parameterized Exploration (PE), a simple family of methods for tuning the exploration schedule which leverages an estimated model of dynamics of the decision problem and accounts for the time horizon. The proposed methods provides significant empirical gains relative to competing methods in variety of empirical examples and an application the management of type I diabetes using mobile interventions. We show that the proposed method is consistent for the optimal exploration schedule within the class under consideration and show how its estimated sampling distribution can be used to design robust and safe exploration strategies.