Learning Fair Contextualized Word Representations with Natural Language Inference

Abstract

Contextualized word representations are the keystone of modern natural language processing. However, the fact that these representations encode latent stereotypes has given rise to much scientific concern, particularly as language models trained on these representations are shown to magnify discriminatory judgements in downstream tasks. Focusing on gender bias as a representative example, we propose NLIDA (Natural Language Inference Debiasing Approach), which is a training procedure that relies upon annotated natural language inference data to generate fair and semantically informative embeddings. Vis-à-vis its state-of-the-art counterparts, NLIDA exhibits fairer performance across a suite of extrinsic measurements spanning occupation classification, natural language inference, and coreference resolution. Furthermore, it retains its capacity for language understanding across two general-purpose language benchmarks. Through systematic ablations and analysis, we find that NLIDA learns a more equitable representation space, such that gendered concepts are pushed together and pulled apart from non-gendered concepts. Together, our results highlight the suitability of leveraging natural language inference as an effective and powerful means of bias attenuation.