Gaussian Differential Privacy, with Applications to Deep Learning

• Activity Number: 538 - Emerging Topics in Private Data Analysis
• Type: Topic Contributed
• Date/Time: Thursday, August 6, 2020 : 1:00 PM to 2:50 PM
• Sponsor: IMS
• Author(s): Jinshuo Dong* and Aaron Roth and Weijie Su
• Companies: University of Pennsylvania and University of Pennsylvania and University of Pennsylvania
• Keywords: differential privacy; central limit theorem; deep learning; machine learning; Blackwell's information ordering; SGD

Abstract

Privacy-preserving data analysis has been put on a firm mathematical foundation since the introduction of differential privacy (DP) in 2006. In this work, we take the statistical decision theoretic perspective of DP and observe a central limit behavior for composition of private algorithms. In order to study this phenomena, we develop a complete toolkit. First, we propose “f-DP”, a relaxation of the traditional parametrization of DP. This generalization allows for neat, lossless and ``algebraic'' reasoning about nearly every important tool in DP literature. We define a canonical single-parameter family of definitions within our class that is termed “Gaussian Differential Privacy” (GDP), based on hypothesis testing of two shifted normal distributions. We prove a privacy central limit theorem, which shows that under mild conditions, DP guarantees of any private algorithm converge to GDP in the limit under composition. We demonstrate the use of these tools by various experimental results of privacy-preserving deep learning, improving on accuracy of the models while providing the same or better privacy guarantee than previous results.