Issues with Training, Testing, and Validation Data Sets in the Development of Diagnostics Devices

• Activity Number: 188
• Type: Contributed
• Date/Time: Monday, August 10, 2015 : 10:30 AM to 12:20 PM
• Sponsor: Section on Medical Devices and Diagnostics
• Author(s): R. Lakshmi Vishnuvajjala*
• Companies: FDA
• Keywords: Validation ; Testing ; Training ; Classifier ; Diagnostic Device

Abstract

Model development and validation are critical parts in the development of classifiers, or diagnostics devices as they are called in submissions to FDA. The integrity of methods used to develop and validate classification models ensures the performance of the diagnostics devices. There is a lot of confusion about training, testing and validation datasets, as well as internal and external validation of models. We will discuss some good practices for developing and validating classification models in diagnostic devices. Specially, we will investigate some problems frequently encountered with training and validation datasets which can lead to overly optimistic estimates of performance metrics.