Characterization of learning algorithms learned by deep meta-reinforcement learning agents

Abstract

In 2016, Wang et al. and Duan et al. demonstrated that meta-learning can emerge in deep neural networks by showing that recurrent neural networks (RNNs) can be trained to implement reinforcement learning (RL) strategies for different families of tasks. This introduction of deep meta-reinforcement learning (deep meta-RL) has had significant implications not only in the field of machine learning but also in the natural domains of neuroscience and behavioral psychology.

Psychologists and neuroscientists are primarily concerned about how meta-learning works in the brain and thus are interested in analyzing interpretable computational models of behavior and cognition. As a result, in order to maximize the utility of deep meta-RL to the fields of neuroscience and behavioral psychology, it is important to attain a mechanistic understanding of the learning strategy learned by deep meta-RL agents (and implemented by the underlying RNNs). In general, understanding the behavior and mechanisms of deep neural networks has been an open problem in the field of machine learning. Although deep neural networks are notoriously difficult to interpret (and even moreso for RNNs), we make tangible progress towards characterizing the underlying RNNs which implement the RL algorithms learned by deep meta-RL agents (meta-RNNs) on a family of bandit tasks. We demonstrate certain learning properties exhibited by these meta-learned RL strategies and elucidate the structure of the hidden state space used by the meta-RNNs. We also show that simple linear approximations (in the form of linear state machines) can be derived from deep meta-RL agents and achieve a high degree of replication accuracy.

Our work has important implications as a bridge between demonstrating that meta-learning strategies can be implemented by computational models (e.g., deep neural networks), and applying these computational models to understanding how meta-learning works in the human brain.