Rethinking Prediction Errors in Dopaminergic Signals in the Brain

Abstract

The hypothesis that midbrain dopamine (DA) neurons broadcast an error for the prediction of reward (reward prediction error, RPE) is among the great successes of computational neuroscience (Houk, Adams, and Barto 1995; Montague, Dayan, and Sejnowski 1996; Schultz, Dayan, and Montague 1997). However, modern empirical results have challenged core aspects of this theory, and my dissertation looks in two such datasets: the first of which delved into how DA responses reflect movement correlates that cannot be explained by a RPE (Parker et al. 2016) and the second of which showed how DA might reflect heterogeneous features rather than convey a scalar, global signal (Engelhard et al. 2019). In my dissertation, I build new models and theories to update the classic RL algorithm TD learning in order to account for these new results. In Chapter 2, I investigate dorsal medial striatum (DMS) projecting DA neurons and how they do not reflect RPE with respect to contralateral movement, but contralateral movement directly and the value of the chosen action. In Chapter 3, I introduce a “feature-specific” PE model in order to explain heterogeneous DA responses in the VTA. I argue that the heterogeneity is a reflection of the input state features upstream of the DA neurons, and show how our model can recapitulate how patterns of heterogeneity might arise for reward prediction errors and movement responses. In Chapter 4, I revisit the dataset from Chapter 2 to determine if the movement correlates could be prediction errors with respect to movements rather than reward. While I could not find an action PE with respect to contralateral movement, I am able to show that DMS-projecting DA population at lever presentation does reflect an identity-free action PE, or an “action-surprise” signal. Overall, my dissertation aims to rethink the PE signal in TD learning, showing that classic RPE accounts of DA need to be updated to account for these new, puzzling data. Together, my models show that DA data may be better understood as a more generalized family of PE models, reflecting a richer signal than just a scalar RPE depending on the inputs they receive.