Online Program

Return to main conference page

All Times EDT

Wednesday, September 22
Wed, Sep 22, 1:00 PM - 2:00 PM
Poster Session I

Bayesian Logistic Regression for Online Recalibration and Revision of Clinical Prediction Algorithms with Safety Guarantees (302405)

View Presentation

*Jean Feng, University of California, San Francisco 
Alexej Gossmann, US Food and Drug Administration 
Romain Pirracchio, University of California, San Francisco 
Berkman Sahiner, US Food and Drug Administration 

Keywords: AI/ML-based SaMD, online learning, Bayesian inference

After a machine-learning based Software as a Medical Device (ML-based SaMD) has been approved by the FDA and deployed, post-marketing data can be used to fine-tune the prediction algorithm and adapt to temporal shifts. Although continually evolving prediction models have the potential to outperform static (or "locked") models, such procedures are prone to overfitting and may lead to model deterioration. In this work, we discuss how one may design safe Algorithm Change Protocols (ACPs), as outlined in the FDA's 2019 discussion paper for regulating modifications to AI/ML-based SaMDs. To this end, we investigate dynamic model recalibration and revision using Bayesian logistic regression (BLR) and Bayesian dynamic logistic regression with a Markov prior for distributional shifts (MarBLR). We derive theoretical guarantees to show that BLR and MarBLR can control online versions of Type I and II error. In simulation studies, BLR and MarBLR were able to recalibrate forecasted risks from an underlying model to new patient populations and within subpopulations, learn linear model revisions, or wrap around black-box model-updating procedures to improve their safety. In a case study on inpatient admission data from UCSF, BLR and MarBLR updated a risk prediction model for COPD diagnoses over a nine-year time span, where they safely shifted their dependence on a locked gradient boosted tree to a continually refitted version. Throughout our empirical analyses, BLR and MarBLR consistently outperformed the locked model. Although there are numerous risks associated with online model updating, our results suggest that continual learning algorithms for AI/ML-based SaMDs can maintain or improve their safety and effectiveness over time if carefully designed ACPs and high-quality data monitoring procedures are employed.