Skip navigation
Please use this identifier to cite or link to this item:
Title: Understanding Deep Learning via Analyzing Dynamics of Gradient Descent
Authors: Hu, Wei
Advisors: Arora, Sanjeev
Contributors: Computer Science Department
Keywords: deep learning
learning theory
machine learning
Subjects: Computer science
Artificial intelligence
Issue Date: 2021
Publisher: Princeton, NJ : Princeton University
Abstract: The phenomenal successes of deep learning build upon the mysterious abilities of gradient-based optimization algorithms. Not only can these algorithms often successfully optimize complicated non-convex training objectives, but the solutions found can also generalize remarkably well to unseen test data despite significant over-parameterization of the models. Classical approaches in optimization and generalization theories that treat empirical risk minimization as a black box are insufficient to explain these mysteries in modern deep learning. This dissertation illustrates how we can make progress toward understanding optimization and generalization in deep learning by a more refined approach that opens the black box and analyzes the dynamics taken by the optimizer. In particular, we present several theoretical results that take into account the learning dynamics of the gradient descent algorithm. In the first part, we provide global convergence guarantees of gradient descent for training deep linear networks under various initialization schemes. Our results characterize the effect of width, depth and initialization on the speed of optimization. In addition, we identify an auto-balancing effect of gradient flow, which we prove to hold generally in homogeneous neural networks (including those with ReLU activation). In the second part, we study the implicit regularization induced by gradient descent, which is believed to be the key to mathematically understanding generalization in deep learning. We present results in both linear and non-linear neural networks, which characterize how gradient descent implicitly favors simple solutions. In the third part, we focus on the setting where neural networks are over-parameterized to have sufficiently large width. Through the connection to neural tangent kernels, we perform a fine-grained analysis of optimization and generalization, which explains several empirically observed phenomena. Built on these theoretical principles, we further design a new simple and effective method for training neural networks on noisily labeled data.
Alternate format: The Mudd Manuscript Library retains one bound copy of each dissertation. Search for these copies in the library's main catalog:
Type of Material: Academic dissertations (Ph.D.)
Language: en
Appears in Collections:Computer Science

Files in This Item:
File Description SizeFormat 
Hu_princeton_0181D_13832.pdf3.63 MBAdobe PDFView/Download

Items in Dataspace are protected by copyright, with all rights reserved, unless otherwise indicated.