Metacognition: toward a computational framework for improving our minds

Abstract

In this dissertation I will show how reinforcement learning (RL) can be applied to the inner workings of cognition. The usual application of RL is to understand human behavior or build intelligent machines interacting in the external world. The same RL formalism can be inverted onto cognitive processes themselves, resulting in a normative account of how to explore and select mental computations, referred to as metacognitive RL. This framework can 1) be used to generate observable behavioral predictions, 2) provide a resource-rational benchmark for both assessing and improving cognition, and 3) motivate cognitive process models based on interacting RL systems. The formalism of metacognitive RL rests on meta-level Markov Decision Processes (meta-MDPs), which provide a general-purpose computational framework that can also make task-specific predictions. The first study applies the resource-rational framework to risky choice resulting in the identification of heuristics and accurate predictions about how people adapt their use of heuristics. The second study uses the same metacognitive RL framework to predict which structures of the environment will enhance metacognitive learning in humans during a planning task. In the third study, rather than manipulate the decision environment---which is often infeasible to do in the real-world---the metacognitive RL framework is used to produce feedback in the form of \textit{pseudorewards}, resulting in faster metacognitive learning in a related planning task. Next, a new cooperative RL architecture is proposed. This approach also uses pseudorewards to promote learning, but rather than generate the pseudorewards from a computational model, it is proposed that they may be produced internally by a distinct RL system. The successful application of the metacognitive RL framework to understand and improve cognitive function depends critically on developing machine learning methods to solve these problems. In the final chapter, I briefly explore the application of a recently proposed machine learning method for solving meta-MDPs.