Exploring multi-armed bandit decision-making strategies in an underwater vehicle testbed

Abstract

The problem of field estimation, or finding the spatial distribution of a resource in space, has many applications in problems of robotic search. This thesis approaches the problem in the framework of a Gaussian Multi-Armed Bandit (MAB) task, a problem in which an agent must learn about an unknown environment while maximizing expected reward. The arms correspond to discretized points in space, and the smoothness of the field is modeled as spatial correlation between the arms. Up-per Confidence Limit (UCL), an algorithm developed by Reverdy et al. in 2014 for Gaussian MAB problems with correlated arms and prior knowledge, is then applied to this problem. The smoothness of the field is measured by a parameter known as the length scale. In real world applications, the agent can only have an estimate of this length scale. This thesis explores the performance of UCL with correlation in comparison to other algorithms when the estimate of the length scale is correct. The effect of overestimates and underestimates in the length scale is then explored for fields of different smoothness. The search task is finally implemented in a testbed with a robot, giving additional metrics of performance. The simulations showed that knowledge of the spatial correlation of the arms can result in improvements in performance using UCL when compared to other algorithms that do not account for correlation. In addition, it is shown that, in the cases studied, best performance is not actually obtained for the correct length scale estimate, but for some particular overestimate. This effect must be studied further, but the results suggest that an agent performing this search should use an overestimate and not an underestimate of the length scale that describes the field, provided this overestimate is not grossly inaccurate.