Using an instruction interface for compiling deep learning applications to the NVIDIA Deep Learning Accelerator (NVDLA)

Abstract

Advances in neural networks are increasingly enhancing or replicating human abilities. Currently, consumers train neural networks and run inference on CPUs and GPUs. Some commercial uses of deep learning accelerators exist and offload supported operations to faster hardware. However, due to specialized drivers and compilers, these are not portable or applicable for widespread use. This thesis extends the D2A methodology beyond the current set of supported accelerators, FlexASR, HLSCNN and VTA, to include the more complex NVIDIA Deep Learning Accelerator (NVDLA). This enables a range of applications written in domain specific languages, such as TensorFlow, to be automatically compiled for the NVDLA and thus makes the NVDLA more widely usable. Using only direct neural network to NVDLA operation matches, 65 of the 69 neural network operations in the popular ResNet-18 image classification model could be offloaded. One additional operator was matched when the semantic rewrite rules of D2A were also included. The accuracy of offloaded individual neural network layers as well as the full direct-matched model was successfully validated against the CPU by using existing NVDLA simulators. Overall, the NVDLA’s large set of supported operators allows for a significant number of accurate offloads in convolutional models. Further, D2A’s combined semantic and accelerator specific rewrites result in more offloaded operators than simple direct matching. This advantage would be larger for less optimally expressed models.