Navigating Physically-Informed Loss Landscapes with Stochastic Gradient Descent

Abstract

Physics-informed neural networks (PINNs) have seen recent success in modeling phenomena governed by differential equations, but there remains much room for theoretical work on ensuring their accuracy. For convenience, PINNs often check whether the governing equation is satisfied only at a finite set of points, approximating the overall loss with a Monte Carlo sum. We find that this can lead to inaccurate predictions, especially in settings where the prediction has a steep gradient. In lieu of the traditional approach, we propose two methods for ensuring the accuracy of predictions. First, we show how with convenient choices of activation functions it is possible to compute the exact loss directly, although this comes at a large computational cost. Second, we find that by resampling the finite set of points at every iteration, PINNs achieve better predictions in a variety of settings, and this incurs only a small computational cost. The set of points can be chosen either completely randomly, as in stochastic gradient descent, or adversarially, building on the recent success of generative adversarial networks. With each approach, we find that the resulting prediction converges to the theoretical optimum, which is computed using techniques from calculus of variations to characterize the landscape of the physics-informed loss function.