Towards the Automatic Discovery of Deep Recurrent Neural Network Architectures

Abstract

Recurrent Neural Networks (RNNs) have become a staple tool for solving sequential learning problems, but hand-designing deep RNN architectures remains a laborious task because of the enormous search space of candidate configurations. DreamCoder is an inductive programming framework that efficiently traverses a vast program search space by building hierarchical symbolic representations of knowledge. Therefore, we posit that DreamCoder can be applied towards the generation of deep RNN architectures. This thesis makes three elementary steps towards this goal. First, we extend the DreamCoder model to the vector algebra domain, culminating in the rediscovery of nonparametric models for multidimensional RNN cells and cell components out of basic arithmetic and list processing primitives. Next, we harness DreamCoder to learn parametric functions in the single-dimensional algebra domain, leading to the rediscovery of parametric models for single-dimensional RNN cell components. Finally, we present a roadmap to extend this thesis’s rediscovery of single RNN cells to the discovery of deep RNN architectures, setting the stage for the generation of novel RNN architectures that have been tailored to model individual sequential learning tasks.