A Survival Reinforcement Learning Framework and Its Biomedical Applications

• Activity Number: 309 - Statistical Reinforcement Learning
• Type: Invited
• Date/Time: Tuesday, August 9, 2022 : 2:00 PM to 3:50 PM
• Sponsor: Section on Statistical Computing
• Author(s): Hunyong Cho* and Shannon T. Holloway and David J. Couper and Michael Kosorok
• Companies: University of North Carolina at Chapel Hill and North Carolina State University and University of North Carolina, Chapel Hill and University of North Carolina at Chapel Hill
• Keywords: reinforcement learning; Q-learning; dynamic treatment regime; survival analysis; random forest

Abstract

We introduce a reinforcement learning approach that optimizes survival outcomes. Motivating examples include finding dynamic treatment strategies that provide a maximal survival time for cancer patients going through a series of chemotherapies. Challenges of using survival data in reinforcement learning settings, e.g., missing information due to early failure or drop-out, are discussed. Our approach generalizes the conventional Q-learning framework in the sense that backward recursion is done on processes, e.g., the optimized survival curves, rather than on scalar values, or the optimized means. We use a modified version of random forest for the state-action value function estimation, and the theoretical properties of the estimator—consistency and rates of convergence—are shown. The improved empirical performances over the existing methods are illustrated through simulations. As biomedical applications, we obtain dynamic treatment regimes for leukemia and cardiovascular patients.