Prediction Error During Conversational Speech: Comparing Deep Language Models and the Human Brain

Abstract

Advances in deep learning have resulted in new deep language models (DLMs) that are trained using a self-supervised next-word prediction task. These models continue to establish benchmarks in the field of Natural Language Processing (NLP), and their performance on tasks such as question answering, summarization, translation, and even text generation have become comparable to human performance. This begs the question: to what extent are the underlying computational mechanisms employed by DLMs and the human language faculty similar? This study uses DLMs to investigate prediction error – a well-documented phenomenon in language neuroscience that arises when a semantic or syntactic expectation does not align with what is heard – using conversational, uncontrolled electrocorticography (ECoG). Previous prediction error work has relied on post word onset event-related potentials (ERPs) produced in highly controlled scenarios to demonstrate a distinct neural signature for words that are semantically or syntactically surprising (e.g., N400 and P600 effects). However, the ecological validity of such designs and the attribution of these effects to mistakes made during active next-word prediction remain topics of debate in the field. In this study I propose that DLMs can be used to track human predictions and consequently allow researchers to evaluate language-based prediction error in naturalistic data that would have once been inaccessible. I find that there is a meaningful difference in the ERPs in response to words that a DLM can predict correctly versus those that are predicted incorrectly during conversational language comprehension, in accordance with previous work. Through a linear encoding model that predicts brain activity from word embeddings over multiple lags, I provide an extension of the prediction error effect, and I contend that this extension substantiates the claim that the observed prediction error effect reflects mistakes made during spontaneous next-word prediction. Finally, to further support these findings, I demonstrate that a similar prediction error effect is not present during speech production and speculate on the role of next-word prediction during production, which until this point has received little to no attention in the field as a result of methodological and paradigmatic constraints. The unique combination of intercranial recordings and naturalistic design implemented in this study offer a window into future work that will push our understanding of the human language faculty beyond the processing of information and toward the conception and expression of that information in the first place.