All Times EDT
Keywords: diagnostic test, deep learning, convolutional neural network, hidden layer, confounding
Machine learning (ML) / artificial intelligence (AI) is increasingly being employed for a panoply of intended uses including screening, diagnosis, prognosis, risk prediction, clinical decision support, and phenotype classification. ML-enabled software as a medical device (SaMD) can present many challenges to regulatory authorities responsible for evaluating their safety and effectiveness. First, training of the model may be inadequate, resulting in lack of generalizability (fairness), i.e., poor performance in some subsets, because of confounding or because some datapoints are highly influential in model fitting. Second, a main benefit of ML resides in its capability to learn from real-world use and experience and therefore to improve its performance as data accumulate, but the regulatory pathway to fulfill this potential in a least burdensome manner needs further development. Other challenges include model transparency and interpretability, use of synthetic data, reuse of validation data, errors in the reference standard (ground truth determination), corruption of performance via introduction of adversarial data, and calibration and uncertainty quantification of risk predictions. After briefly reviewing US FDA and other regulatory initiatives supporting the development and validation of SaMD's, I’ll review some of the statistical challenges with ML-enabled medical devices and respond to the previous talks.