A Quantitative Analysis of Prompt Engineering and In-Context Learning

Abstract

Prompt Engineering has become the most accessible entry point into Large Language Models for both the Public and even Academic Researchers. With the advent of large privatized language models, fine-tuning language models becomes computationally intractable, and even contractually impossible. Given the impressive few-shot capabilities of these large language models, a much more reasonable task-priming method is prompting. While research has yielded very powerful but specific ideas such as Chain of Thought Prompting [10], and Low Perplexity Instructions [4], there is very little work studying general trends in prompt engineering. This thesis represents my year-long journey through this task. First I study few-shot demonstrations and ask the question: Can we predict whether a prompt will lead to good performance just by looking at the model? I am able to successfully train a prompt performance prediction model that generalizes to unseen tasks in the same domain. Next, I focus on instructions, and propose a formal benchmark to evaluate instructions and prompt templates on in-context learning metrics. We find several nuances that have not been explored by state of the art work.