Exploding and Vanishing Gradients in High-Channel Convolutional Neural Networks

Abstract

We analyze the behavior of the moments of the gradient in a randomly initialized convolutional neural network C with ReLU activations. We show that in randomly initialized networks where the variance of the weights is inversely proportional to the number of channels and kernel sizes, it is possible to avoid the Exploding-Vanishing Gradient problem by scaling the number of channels as the network depth increases; however, the growth rate needed to control the weak bounds we derive for the mo- ments of the gradient is too large to be feasible in practice.