Evaluation of Disinformation Detection Systems Over Time Using Content Based Features

Abstract

The proliferation of online disinformation is a serious threat to economic, political, and public discourse. Existing approaches to mitigate spread either rely on manual analysis and fact checking, which is neither scalable nor efficient, or on automated approaches that analyze network and content level details to classify disinformation on the article or domain level. To better facilitate work in domain level detection, this study examines how model performance changes over time using content based features. We develop a system that crawls domains for articles and create a comprehensive dataset of 516 disinformation and news domains from 2010 to 2019. We generate content based features for each domain and train models over different time periods. We analyze what features are more important in differentiating between disinformation and news domains and show that, as time progresses, a subset of features are time invariant and prevent decreases in model performance over time