Optimization Inspired Neural Networks for Multiview 3D Reconstruction

Abstract

3D reconstruction is usually formulated as an optimization problem. We define a measure of solution quality (objective) and design a search algorithm for finding good solutions (optimizer). Both problems are challenging. The objective needs to adequately capture the complexity of the 3D world while simultaneously being simple enough such that it is tractable using standard techniques. This is in contrast to deep learning where inference is performed using a neural network with learned weights. Despite the success of neural networks on semantic tasks their performance on multiview reconstruction tasks is often less accurate and less generalizable than approaches based purely in optimization. In this thesis, we show that optimization algorithms can be learned from data and used for motion estimation and reconstruction. We design a generic network architecture with interleaves iterative updates with implicit layers. We show that such networks can automate both modeling (objective function design) and optimization. We apply this approach to a three representative multiview problems: optical flow, scene flow, and simultaneous localization and mapping. We push forward performance on these problems while using a general design strategy.