Model and Data Efficiency in Deep Learning

Abstract

Deep learning has achieved great and broad breakthroughs in numerous real-world applications with the advancement of larger size models and the explosive growth and availability of data. How- ever, the deep learning models usually have excessive computational and memory cost that are not friendly to practical deployment on mobile or edge devices. Moreover, the deep learning models face challenges in learning and adapting rapidly from only a few examples to solve new tasks. Hence, this thesis proposes techniques to learn computationally efficient model architectures and methods to improve the few-shot learning ability. We start with subspace analysis methods with application to the feature selection problem. We then extend these methods to deep neural network structural learning (SL), with the objective of reducing the redundant parameters to obtain the optimal down-sized model that can retain or even improve the accuracy. More efficient SL method based on the hybrid pruning-regrowing technique and more generalized SL method that can reduce the model across many more dimensions are also introduced. Going beyond static model designs, we also present dynamic neural network approaches that can adapt the model weights and architectures to different inputs on-the-fly during inference to control the computation efficiency and improve the representation ability. Apart from model efficiency, we also present techniques to train models that can rapidly generalize from a few examples. We propose a few-shot architecture adaption method to customize task-specific model structure for diverse few-shot tasks by meta-learning a task-aware architecture controller. Different from the traditional NAS methods that require a separate search cost on each new task, our method directly generates the task-specific model structure from the dataset in GPU minutes after a one- time meta-training cost. Finally, we propose a cross-modality self-supervised learning framework by masked image pretraining on language assisted representations. The resulting models produce high quality transferable representations that advance the accuracy on numerous computer vision tasks and demonstrate strong robustness to adversarial/out-of-distribution samples. Moreover, the resulting models are amenable to structural learning for greater computation efficiency gains and to low-resource task adaptation for better data efficiency.