Skip navigation
Please use this identifier to cite or link to this item:
Title: Meta-Reinforcement Learning with Episodic Recall: An Integrative Theory of Reward-Driven Learning
Authors: Ritter, Samuel
Advisors: Botvinick, Matthew
Contributors: Neuroscience Department
Keywords: deep learning
episodic memory
model-based learning
model-free learning
reinforcement learning
working memory
Subjects: Neurosciences
Computer science
Cognitive psychology
Issue Date: 2019
Publisher: Princeton, NJ : Princeton University
Abstract: Research on reward-driven learning has produced and substantiated theories of model-free and model-based reinforcement learning (RL), which respectively explain how humans and animals learn reflexive habits and build prospective plans. A highly developed line of work has unearthed the role of striatal dopamine in model-free learning, while the prefrontal cortex (PFC) appears to critically subserve model-based learning. The recent theory of meta-reinforcement learning (meta-RL) explained a wide array of findings by positing that the model-free dopaminergic reward prediction error trains the recurrent prefrontal network to execute arbitrary RL algorithms -- including model-based RL -- in its activations. In parallel, a nascent understanding of a third reinforcement learning system is emerging: a non-parametric system that stores memory traces of individual experiences rather than aggregate statistics. Research on such \textit{episodic learning} has revealed its unmistakeable traces in human behavior, developed theory to articulate algorithms underlying that behavior, and pursued the contention that the hippocampus is centrally involved. These developments lead to a set of open questions about (1) how the neural mechanisms of episodic learning relate to those underlying incremental model-free and model-based learning and (2) how the brain arbitrates among the contributions of this abundance of valuation strategies. This thesis extends meta-RL to provide an account for episodic learning, incremental learning, and the coordination between them. In this theory of episodic meta-RL (EMRL), episodic memory reinstates activations in the prefrontal network based on contextual similarity, after passing them through a learned gating mechanism (Chapters 1 and 2). In simulation, EMRL can solve episodic contextual water maze navigation problems and episodic contextual bandit problems, including those with Omniglot class contexts and others with compositional structure (Chapter 3). Further, EMRL reproduces episodic model-based RL and its coordination with incremental model-based RL on the episodic two-step task (Vikbladh et al., 2017; Chapter 4). Chapter 5 discusses more biologically detailed extensions to EMRL, and Chapter 6 analyzes EMRL with respect to a set of recent empirical findings. Chapter 7 discusses EMRL in the context of various topics in neuroscience.
Alternate format: The Mudd Manuscript Library retains one bound copy of each dissertation. Search for these copies in the library's main catalog:
Type of Material: Academic dissertations (Ph.D.)
Language: en
Appears in Collections:Neuroscience

Files in This Item:
File Description SizeFormat 
Ritter_princeton_0181D_12836.pdf2.79 MBAdobe PDFView/Download

Items in Dataspace are protected by copyright, with all rights reserved, unless otherwise indicated.