All Times EDT
Keywords: Machine learning, hypothesis testing, online learning, statistical process control, quality improvement, quality assurance, data streams, performance monitoring
Machine learning (ML) and artificial intelligence (AI) algorithms have the potential to derive insights from clinical data and improve patient outcomes. However, these highly complex systems are sensitive to changes in the environment and liable to performance decay. Even after their successful integration into clinical practice, ML/AI algorithms should be continuously monitored and updated to ensure their long-term safety and effectiveness. In this talk, we will discuss how sequential monitoring procedures from statistical process control can be adapted to detect performance decay in ML algorithms. We will emphasize the unique challenges faced when monitoring ML-based clinical decision support systems in particular, as these systems can modify the very outcomes that they aim to predict. Then turning our attention to model updating, we describe how online testing and learning frameworks can be used to safely approve and generate modifications to ML algorithms, respectively. By comparing the merits and limitations of these different frameworks, we aim to bring attention to open questions on the maintenance of AI/ML systems over time.