Exploiting Program Representation with Shader Applications

Abstract

Programs are widely used in content creation. For example, artists design shader programs to procedurally render scenes and textures, while musicians construct “synth” programs to generate electronic sound. While the generated content is typically the focus of attention, the programs themselves offer hidden potential for transformations that can support untapped applications. In this dissertation, we will discuss four projects that exploit the program structure to automatically apply machine learning or math transformations as if they were manually designed by domain experts. First, we describe a compiler-based framework with novel math rules to extend reverse mode automatic differentiation so as to provide accurate gradients for arbitrary discontinuous programs. The differentiation framework allows us to optimize procedural shader parameters to match target images. Second, we extend the differentiation framework to audio “synth” programs so as to match the acoustic properties of a provided sound clip. We next propose a compiler framework to automatically approximate the convolution of an arbitrary program with a Gaussian kernel in order to smooth the program for visual antialiasing. Finally, we explore the benefit of program representation in deep-learning tasks by proposing to learn from program traces of procedural fragment shaders – programs that generate images. In each of these settings, we demonstrate the benefit of exploiting the program structure to generalize hand-crafted techniques to arbitrary programs.