The factor structure of reinforcement learning behaviors

Abstract

Reinforcement learning (RL) is a cognitive and algorithmic framework that specifies how agents (e.g., humans, animals, machines) can learn reward-maximizing policies through trial-and-error. The burgeoning field of computational psychiatry has demonstrated the relevance of RL for understanding psychiatric conditions including depression, obsessive- compulsive disorder, and substance use. However, the field is currently hampered by many theoretical and psychometric issues that limit the validity, interpretability, and utility of symptom-behavior correlations. In this dissertation, I develop a battery of reliable RL tasks — spanning multiple dimensions of reinforcement learning behaviors including exploration, risk sensitivity, Pavlovian-instrumental biases, and model-based planning — in order to characterize the factor structure of reinforcement learning behaviors. The ultimate aim of this research is to better understand the dimensionality of individual-differences in RL behaviors, and their relationship to cognitive ability and psychiatric symptoms. In Chapter 2, I introduce a novel task measure of differential sensitivity to signed prediction errors that is test-retest reliable and robust against confounding from choice autocorrelation. In Chapter 3, I introduce an improved measure of Pavlovian-instrumental biases that is test-retest reliable and less subject to practice effects. In Chapter 4, I validate the matrix reasoning item bank and use it to construct brief, reliable measures of fluid reasoning. In Chapter 5, I show how inattentive responding by participants can induce spurious symptom-behavior correlations. I then validate a screening method which prevents them. Finally, in Chapter 6, I investigate the factor structure of individual-differences in performance on the RL task battery in a sample of N=975 participants. I show that prior research has underestimated across-task behavioral correlations by neglecting measurement error. I also demonstrate that covariation in RL behaviors are well-described by a hierarchical factor model, in which behavior reflects both a general factor — capturing the overall acuity of participants’ decision-making — and a number of specific factors that reflect participants’ proclivities towards task-specific strategies. I also find that only the general factor correlates with participants’ cognitive ability and psychiatric symptoms. In the conclusion, I interpret the results with respect to the current state of computational psychiatry and discuss a number of challenges for the field moving forward.