Adaptive Confidence Intervals for Regression Coefficients in Q-Learning

• Activity Number: 3
• Type: Invited
• Date/Time: Sunday, August 1, 2010 : 2:00 PM to 3:50 PM
• Sponsor: General Methodology
• Author(s): Eric B. Laber and Min Qian and Susan Murphy*+
• Companies: University of Michigan and University of Michigan and University of Michigan
• Keywords: dynamic treatment regimes ; adaptive treatment strategies ; reinforcement learning ; non-regular

Abstract

Dynamic treatment regimes (or treatment policies) are used to operationalize multi-stage decision making in the medical field. Common approaches to constructing the dynamic treatment regimes from data, such as Q-Learning, employ non-smooth functionals of the data. The non-smoothness leads to non-regular asymptotics under certain generative models. Methods that ignore the non-regularity have poor performance in small samples. In this talk, we present a bootstrap based method for constructing asymptotically valid confidence sets. This method is adaptive in the sense that it provides exact coverage when the true underlying generative model leads to regular asymptotics and is conservative otherwise. Empirical studies show that the amount of conservatism is small.