Optimizing Digital Diagnostic/Prognostics Algorithms: A New Framework and Approach Beyond Area Under the Receiver Operating Characteristic Curve

• Activity Number: 361 - Using Statistical Foundations to Demonstrate Effectiveness of ML/AI Algorithms for Clinical Utility
• Type: Invited
• Date/Time: Wednesday, August 10, 2022 : 8:30 AM to 10:20 AM
• Sponsor: Biopharmaceutical Section
• Author(s): Stephen J Ruberg*
• Companies: Analytix Thinking, LLC
• Keywords: sensitivity; specificity; area under the curve; prevalence; positive predictive value; utility

Abstract

A recent FDA Workshop on AI in medical devices noted that they should be safe and effective, fair for patients of different races, genders and geographies and documented to improve patient outcomes. These principles are aligned with the development of new medicinal products by the pharmaceutical industry. However, FDA’s regulation of AI stands in stark contrast to its oversight of new drugs, where adequate and well-controlled clinical trials (AWCT) are the standard. This talk will draw learnings from Phase 2 drug development and consider how statistical principles outlined in the international standard ICH E9 Statistical Principles for Clinical Trials can be applied to assessing whether AI-enabled devices/tools (I.e., interventions) meet a standard of safe and effective. Just as Phase 2 trials are used to explore and optimize dosage and identify appropriate patient populations for a new medicine, this talk will propose a broad framework for optimizing digital diagnostic/prognostic algorithms through AWCT. That framework goes beyond the usual AUC metric used by AI tool developers so that substantial evidence of efficacy and safety of an AI tool can be demonstrated rigorously.