Agents with aspirations: A Prospect Theory approach to intrinsic motivation in Reinforcement Learning

Abstract

For centuries, philosophers have debated the ways in which human minds should optimally manage the trade-off between achieving long-term external goals and maintaining immediate internal happiness. Various different approaches have been taken, from perspectives which prioritize being content with one’s lot to ambitious calls for setting high aspirations. When investigated in the context of intrinsically motivated reinforcement learning (RL), the question of how we should optimally manage our motivations yields interesting results which shed light on the optimal behavior of both humans and machines. This thesis explores the effect of shaping the reward function of an RL agent in order to investigate what forms of intrinsic motivation improve an agent’s ability to learn in different environments. By testing two reward functions based on Dubey et al.’s aspiration model and Kahneman and Tversky’s Prospect Theory, I discover a new reward function which I call the Prospect Theory reward function that performs optimally in dense, sparse and complex environments such as the “Cliff Walking” environment from Sutton and Barto. The parameters of the reward functions are tuned with Bayesian optimization and the performance of the resulting agents is evaluated by comparing the average cumulative reward achieved by each agent. Results reveal that the Prospect Theory value function is a good form of intrinsic motivation to use in multiple categories of environments, suggesting that future RL architectures would benefit from incorporating Prospect Theory into their reward functions.