Towards Robust Models in Deep Learning: Regularizing Neural Networks and Generative Models

Abstract

Deep neural networks are widely used in signal processing from a broad range of areas due to their good performances, including computer vision, natural language processing, automatic driving, and so on. However, people notice that neural networks are easily fooled by adversarial attacks and very sensitive to certain data-related scenarios, such as imbalanced classes and outliers. In this thesis, we focus on enhancing model robustness of deep neural networks from different data distributions.In the first part, we focus on datasets whose distributions are biased naturally, from data collection or the nature of data. We define novel information-entropy-based classification loss functions (entropy weight and entropy noise) to distinguish the difficulty of each sample prediction by either weighting or introducing stochastic noise on top of the cross entropy loss. To evaluate the effectiveness of each loss function, we test the new loss functions on crafted noisy and imbalanced datasets based on MNIST. To illustrate their effectiveness in real scenarios, we show improvements on tasks including computer vision and natural language understanding, compared to the corresponding state of the art (SOTA) models. The results show that models trained with entropy-based loss functions surpass the SOTA models. Deep neural networks have also been demonstrated to be vulnerable to adversarial attacks, where small perturbations intentionally added to the original inputs can fool the classifier. In the second part, we propose Path-Norm regularization to improve robustness of neural networks against adversarial attacks in various Lp norms. By adding Path-Norm regularization, models achieve comparable performance as the SOTA defense methods, and outperform SOTA methods when attacks and training samples are from different Lp spaces. We also introduce Featurized Bidirectional Generative Adversarial Networks (FBGAN), which extracts semantic features of inputs and filters the non-semantic perturbations. FBGAN is pre-trained on clean datasets in an unsupervised manner, adversarially learning a bidirectional mapping between the high-dimensional data space and the low-dimensional semantic space. After the bidirectional mapping, the adversarial data can be reconstructed to denoised data, which could be fed into any pre-trained classifier. We empirically show the quality of reconstruction images and the effectiveness of defense.