A Two-Stage Approach to Evaluate Predictive Accuracy of Deep Neural Networks

• Activity Number: 517 - Deep Learning: Advances and Applications
• Type: Contributed
• Date/Time: Wednesday, July 31, 2019 : 10:30 AM to 12:20 PM
• Sponsor: Section on Statistical Learning and Data Science
• Author(s): Georgianna Campbell* and Emily Nystrom and Hunter R. Lake
• Companies: Naval Information Warfare Center Atlantic and Naval Information Warfare Center Atlantic and Naval Information Warfare Center Atlantic
• Keywords: Deep neural networks; computer vision; two-stage; prediction reliability; machine learning; classification errors

Abstract

Deep neural networks (DNN) are multi-layered neural networks trained on a given dataset to develop a model for predicting future, unlabeled instances. The predictions given by a DNN often do not provide insight about the reliability of the returned predictions. For this research, a two-stage approach will use a previously developed DNN as the first classifier followed by a second DNN to determine the predictive accuracy of the initial model. The goals of this work includes studying prediction reliability and better understanding the underlying characteristics of data that is misclassified. The former effort focuses on utilizing the predictions of previously developed DNNs as input to a second DNN. By extracting the predicted classifications, a new dataset is generated with the labels of either “correctly classified” or “misclassified” according to the initial model. This new data set will be used to train a second DNN to provide insight to end users about the reliability of classifier performance. Our second goal is to evaluate the characteristics of data that are correctly and incorrectly classified to better understand the decision making process of a DNN.