PS43: From FDA Draft Guidance to Practice: Credible and Fit-for-Purpose AI in Clinical Drug Development and Post-Market Surveillance

Artificial intelligence (AI) is evolving into a critical infrastructure for modern drug development, enabling a continuum from insight generation to supporting decision-making, spanning pre-clinical research, clinical trials, and post-market safety monitoring. In January 2025, the U.S. Food and Drug Administration (FDA) issued a draft guidance, "Considerations for the Use of Artificial Intelligence to Support Regulatory Decision-Making for Drug and Biological Products," underscoring both the promise of AI and the need for credible and fit-for-purpose use grounded in transparency, validation, and regulatory fitness. This guidance highlights the growing importance of developing AI methods that can meaningfully support regulatory decision-making while meeting expectations for credibility, robustness, and interpretability. This session features four invited speed talks presenting concrete, regulator-relevant use cases that demonstrate practical and actionable applications of AI across the product lifecycle, followed by a panel discussion with leaders from the FDA, academia, and industry. By pairing detailed case studies with broader regulatory and methodological perspectives, the session aims to bridge innovation and implementation. Session Talks: From Prediction to Inference: Multimodal AI Models and Treatment Effect Estimation in RCTs (Dr. Ting Ye, University of Washington): This talk presents the development and application of multimodal AI models for predicting clinical events and expensive-to-measure biomarkers, alongside a novel statistical framework for leveraging AI-generated predictions to improve treatment effect estimation efficiency in RCTs. Statistical Evaluation Framework for Generative AI in Drug Development (Dr. Sheraz Khan, Pfizer): Integrating generative AI into drug development demands evaluation frameworks that go beyond traditional machine learning metrics. This talk details a comprehensive statistical approach for assessing generative AI applications ranging from regulatory document authoring to patient-facing conversational agents. Key topics include the limitations of deterministic metrics in open-ended tasks, a hybrid methodology combining embedding-based metrics with risk-stratified subject matter expert assessment, simulation-based power analysis using Bayesian ordinal mixed-effects models for non-inferiority testing, and approaches for quantifying inter-rater reliability and human-AI alignment. TBD (Dr. Jamal T. Jones, Office of Surveillance and Epidemiology, FDA Center for Drug Evaluation and Research) Details to come. TBD (Dr. Seth Roberts, BEST IM Task Lead, MITRE) Details to come. Panel discussion: An expert panel discussion will bring together leaders from the FDA, academia, and industry to reflect on their respective perspectives on credible and practical use of AI across multiple dimensions, including operational, design and statistical efficiency, insight generation and decision support, and post-market surveillance. Overall, this session aims to move beyond AI prediction accuracy to focus on how AI-generated insights can be used reliably and appropriately to address key challenges in drug and biological product development and regulatory decision-making.