Identifying GPT: First Principles for Generative AI Detection

Abstract

The safeguards for new technologies to be adopted responsibly need to be released immediately. In response to the mass adoption of generative AI technologies, this project outlines principles for detecting AI generations based on distributional differences in sentence-based perplexities of machine generations from human writing. We also develop and release a novel dataset of human and machine generated articles for analyzing these differences and demonstrate this dataset can be applied to train an effective low-cost AI detector. In addition to perplexity distributions, we introduce another distinction between human and machine writing based on variance in perplexities defined as ‘burstiness’ and posit that it is a quality innate to human writers that will remain a long-term indicator of human writing even with the continued evolution of generative LLMs.