Understanding the Spectral Bias of Deep Learning through Kernel Learning

Abstract

It has been shown empirically that neural networks trained with gradient descent learn simpler functions first. We consider several theoretical justifications of this phenomenon by relating gradient descent to kernel gradient descent through the neural tangent kernel (NTK) and subsequently considering the spectral decay of the NTK.We then consider a setting beyond the lazy regime in which we can approximately describe the discrete evolution of the NTK during neural network training. We use this result to discuss properties of the evolved NTK.