Open-source Neural Photo-Finishing

Abstract

Image processing pipelines are essential for a variety of scientific, commercial, and creative tasks. They take many forms, from camera image signal processors (ISPs) embedded directly into firmware, to standalone software tools for editing captured images.

Traditionally, these pipelines are a tremendous engineering effort, with complicated algorithms developed for each individual processing operation. Crucially, these algorithms are almost always not differentiable, which prevents them from being optimized in machine learning pipelines. Alternatively, image-to-image neural networks have proven quite successful on a variety of image processing tasks (style transfer, editing, semantic understanding), leveraging their differentiable architectures to perform well without extensive software development work. However, these approaches also have limited use cases, as they map directly from input pixel to output pixel; this does not give end-users any manual control over the look of the output image and lacks the requisite inductive bias to adequately model their algorithmic counterparts.

Recent work has struck a powerful balance between contemporary neural networks and legacy algorithmic pipelines, learning individual neural network proxies for each image processing operation and chaining these proxies together. This allows for the differentiable modelling of non-differentiable image processing pipelines with better accuracy and utility than previous methods. However, this work only learned the Adobe Camera Raw pipeline, which is completely closed-source. Moreover, it does not necessarily explore applications to more complicated image processing operations or larger parameter spaces. This thesis proposes an open-source library of tools to learn any image processing pipeline, builds upon previous methods to model more complex image processing operations, and validates the effectiveness of these methods by testing them on Darktable, an open-source image processing pipeline.