Extricating hybrid hypotheses: a comparative analysis of two-step task modeling algorithms and assumptions

Abstract

A long line of literature has established the coexistence of two paradigms during the learning of choice preferences: the model-based system, which computes values based on an internal model of the task environment, and the model-free system, which updates values based solely on experience. Confronted with a recent influx of reinforcement learning models by which to represent this system, we sought to examine the mounting tension in the literature between the highly validated and frequently utilized hybrid model, as originally implemented by Daw, Gershman, Seymour, Dayan, and Dolan (2011), and the newer eligibility adjustment (EA) model, by Toyama, Katahira, and Ohira (2017 & 2019), which differs primarily in its implementation of an altered eligibility update and weighting system that relies on model-free learning as the base system and model-based learning as its modulator. Additionally, we explored the performance of a recent model-based alternative, the transition-dependent learning rates (TDLR) model by DaSilva and Hare (2020). Employing the two-step task developed by Daw et al. (2011) and variants of the hybrid, EA, and TDLR models differing in their added assumptions, we provide robust evidence for the necessity of including perseveration and a forgetting processes in two-step task models in a large participant aggregate (N=1409, from Gillan et al. 2016). Furthermore, we provide tentative evidence to the hypothesis that the EA model augmented by perseveration and forgetting better captures two-step task participant behavior than the augmented hybrid model. Furthermore, while we could not replicate the negative correlations between obsessive-compulsive (OC) tendencies and the weighting parameter w, the forgetting rate F, or the second stage learning rate 2 while controlling for the necessary covariates of age, gender, and IQ, we did find that the inverse temperatures parameters β1 and β2 were strong negative predictors of OC tendencies. Consequently, we argue that obsessive-compulsive participants may employ idiosyncratic strategies that current two-step task models are not able to capture as effectively as evidenced by discrepancies in high OC participants’ model fits.