A Comparison of Q- and A-Learning Methods for Estimating Optimal Treatment Regimes

• Activity Number: 232
• Type: Contributed
• Date/Time: Monday, August 1, 2011 : 2:00 PM to 3:50 PM
• Sponsor: Biometrics Section
• Author(s): Phillip Joel Schulte*+ and Marie Davidian and Anastasios Tsiatis
• Companies: North Carolina State University and North Carolina State University and North Carolina State University
• Keywords: Treatment regimes ; Bias-variance tradeoff ; Model misspecification ; Advantage learning ; Q-learning

Abstract

In clinical practice, physicians must make a sequence of treatment decisions throughout the course of a patient's disease based on evolving patient characteristics. At key decision points, there may be several treatment options and no consensus regarding which option is best. An algorithm for sequential treatment assignment at key decision points, based on evolving patient characteristics, is called a treatment regime. The statistical problem is to estimate the optimal regime which maximizes expected outcome. Q- and A-reinforcement learning are two methods that have been proposed for estimating the optimal treatment regime. While both methods involve developing statistical models for patient outcomes, A-learning is more robust, relaxing some assumptions. However, this additional robustness comes at a cost of increased variability and a bias-variance tradeoff between Q- and A-learning. We explore this tradeoff through parameter estimation and expected outcome for the estimated optimal treatment regime under various scenarios and degrees of model misspecification. We first consider a single treatment decision point for simplicity and then extend to multiple decision points.