Adventures in Synthesis: An Empirical Survey in Using Large Language Models to Train Themselves

Abstract

Self-improving artificial intelligence (AI) systems are the stuff of start-up dreams and sci-fi nightmares. This paper surveys the use of synthetic data in large language models (LLMs), currently the most generally capable artificial intelligence systems. It discusses practical and theoretical problems with fine-tuning a model on synthetic data, particularly for skills like translation. Finally, it reviews the state of the AI industry in the early 2020s and considers trade-offs between general and specialized AI systems.