Understanding and Measuring Privacy Risks in Machine Learning

Abstract

Machine learning models have achieved great success and been deployed prominently in many real-world applications such as face recognition, voice assistants, and recommendation systems. Central to all these machine learning systems is users’ data. However, the sensitive nature of individual users’ data has also raised privacy concerns. A recent thread of research has shown that a malicious adversary can infer private information of users’ data by querying target machine learning models. In this dissertation, we aim to thoroughly understand and measure privacy risks in machine learning, with a specific focus on membership inference attacks where the adversary guesses whether an input sample was used to train the model or not. We first provide a systematic evaluation of membership inference privacy risks by designing benchmark attack algorithms to estimate aggregate privacy risks, and proposing a fine-grained privacy analysis to estimate each individual sample’s privacy risk. Next, we explore the impact on privacy risks of robust training algorithms, which are designed to increase model robustness against input perturbations. We find that these robust training algorithms indeed make machine learning models more vulnerable to membership inference attacks, highlighting the importance of jointly thinking about privacy and robustness in machine learning. Then, we focus on machine unlearning, which is recently proposed to protect users’ data privacy by requiring the machine learning service provider to remove a user’s data from trained models upon its deletion requests. Based on backdooring machine learning models by editing some of training samples, we propose the first verification mechanism which enables individual users to verify whether the service provider follows the deletion request or not. Finally, we present how to leverage privacy attacks to quantify the practical privacy risks ofmachine learning models trained with theoretical upper bound guarantees on privacy leakage. In summary, this dissertation helps the research community to better understand privacy risks in machine learning and provide directions to design more private and trustworthy machine learning models.