West Coast Ballroom
Approval policies for modifications to Machine Learning-Based Software as a Medical Device (307853)
Scott Emerson, University of Washington*Jean Feng, University of Washington
Noah Simon, University of Washington
Keywords: Machine learning, Regulatory Science, Online hypothesis testing, Group sequential designs
Successful deployment of machine learning algorithms in healthcare requires careful assessments of their performance and safety. To date, the FDA has established policies for approving algorithms that are locked prior to marketing where future updates require separate premarket reviews. However, the promise of machine learning algorithms is their ability to learn from a growing body of data and improve over time. Regulating evolving algorithms is a difficult open question: How can we design a policy that approves updates as quickly as possible, but only those that are beneficial? We propose treating the design of approval policies as an online hypothesis testing problem and evaluate various approval policies. The simplest policy, performing a sequence of superiority trials, is insufficient since the multiplicity of hypotheses can lead to bio-creep. To this end, we propose a new group-sequential alpha-investing procedure that controls the expected proportion of bad approvals over time. We demonstrate in simulations and data analyses that our approval policy efficiently approves good model updates while appropriately controlling the online error rates.