On the Robustness of Ensembled Membership Auditing and an Extension to the Zero-Calibration Setting

Abstract

Dataset auditing for machine learning (ML) models is essential for privacy-sensitive applications, such as handling patient data or confidential government information. We explore baseline improvements to the gold-standard Ensembled Membership Auditing (EMA) algorithm, which leverages Membership Inference Attacks (MIA) to identify the training datasets of ML models. We then analyze the robustness of EMA in the presence of proposed defense methods of MIA, including dropout, model pruning, and MemGuard. Our findings reveal that EMA is robust given a reasonable classification loss budget. However, auditing defended models yields a novel False Negative Error Pattern that occurs on large dataset sizes. Our analysis is further stratified across MNIST, Location, and COVIDx datasets, from which we identify convolutional networks as more challenging to audit, especially when exposed to regularization. Additionally, we introduce a metric-set analysis and demonstrate that removing the negative entropy metric improves EMA performance. Lastly, we introduce EMA-Zero, a GAN-based extension to scenarios where a calibration dataset is confidential or unavailable. Remarkably, we find that EMA-Zero with synthetic calibration data generated from as few as 100 samples approaches the EMA baseline. Our research improves the efficacy and interpretability of SOTA data auditing techniques, encouraging novel applications of privacy-preserving ML.