From Learning to Optimal Learning: Understanding the impact of overparameterization on features of neural networks to optimal learning of expensive, noisy functions using low-dimensional belief models

Abstract

This thesis investigates two distinct but important topics in machine learning and optimization.The first topic compares the features of overparameterized neural networks to smaller networks. Chapter 2 presents a methodology to compare the expressivity of features of overparamaterized networks to those of smaller networks. Using this methodology, it finds that smaller networks cannot fully capture the features of over- parameterized networks, and these features are responsible for the superior perfor- mance of overparameterized networks. The chapter also demonstrates through a toy problem that certain features can only be learned by overparameterized networks. The second topic of the thesis is focused on the optimization of costly black-box functions with limited evaluations. In Chapter 3, a new policy called the KGLQ policy is proposed, which approximates the true function locally using a quadratic function and incorporates structural bias by modeling it as heteroscedastic noise that is different from the measurement noise. This approach addresses issues that arise when a value of information policy is used in the presence of parametric models. The KGLQ policy performs competitively compared to existing policies for small budgets, as demonstrated through several test problems evaluated in the chapter. Chapter 4 introduces a global belief model that leverages the concept behind KGLQ. A hierar- chical belief model is developed to produce an approximation by taking into account various levels of estimation from the global belief model. Using this hierarchical model, the HKGLQ policy is developed and shown to be asymptotically convergent. Experiments on test problems provide insights into the performance of KGLQ under larger budgets compared to an asymptotically convergent policy.