Expanding feature-specific to projection-specific models of dopamine heterogeneity with actor-critic methods

Abstract

Midbrain dopamine neurons play an important role in regulating key brain functions, including motor performance, action selection, motivation and reward processing, and working memory. A widely accepted theory in the field of neuroscience is the reward prediction error hypothesis of dopamine, where the phasic activity of dopamine neurons encodes a reward prediction error—defined as the difference between received and predicted rewards—in order to guide learning. However, recent empirical studies of dopamine have reported a functional and spatial heterogeneity across dopaminergic projection targets and within dopamine neuron populations that cannot be easily explained by a scalar RPE. In this thesis, we revisit a feature-specific RPE model by Lee et al. (2022) that explains neuron-specific heterogeneity well but does not fully account for the larger projection-specific differences between movement-related and sensory- related dopamine signals across distinct striatal subregions. We suggest that these projection-specific differences may be accounted for using the actor-critic model, a popular reinforcement learning (RL) model of basal ganglia function in which learning is subdivided into an actor responsible for executing a policy and a critic that evaluates that policy. We hypothesize that these projection- specific differences may reflect functional differentiation between the actor and critic modules. Using a deep reinforcement learning model on a navigation and decision- making task, we test this by explicitly recapitulating the underlying anatomy through separating the final feature layer in the network model of Lee et al. (2022) into distinct actor and critic feature layers and studying whether the learned features and feature RPEs exhibit a preference for movement or sensory variables. Consistent with the feature-specific model findings, we found that heterogeneity arises from feature-based reward prediction, where actor and critic RPEs have heterogeneous and specialized responses to task variables during the cue period while having relatively homogeneous responses to reward during the outcome period. For both actor and critic features, there is a significant difference between movement and sensory variables. Furthermore, we show an overall trend of actor features being better characterized by movement variables and critic features being better characterized by sensory variables, though these effects fail to pass hypothesis testing. Modeling the various levels of heterogeneity within midbrain-striatal dopaminergic circuits is important to understanding dopamine’s diverse roles in adaptive behavioral states, which is central to treating diseases linked to dopamine.