Repainting History: A Framework for Subject-Specific Image Generation in Historical Painting Styles

Abstract

In the cultural and creative industries, one demonstrated use case for AI is to engage art museum visitors through immersive and interactive exhibits. One such exhibit is the SMK Transformerbot at the Statens Museum for Kunst (SMK) in Copenhagen, Denmark, which used generative AI to adapt photographs of museum visitors to historical painting styles. In the exhibit's outputs, however, the faces of the individuals were often disfigured and did not resemble the individuals in the photographs. This directly hindered the exhibit's goal of increasing visitor engagement, and risked offending and alienating visitors. Inspired by this exhibit, this thesis aims to devise a scalable framework for adapting photographs of museum visitors to historical painting styles while preserving the facial likeness of the individuals in the photographs. Building upon the museum's model, our proposed framework uses Stable Diffusion, LoRA, T2I-Adapters, and face restoration models to achieve this task. Through evaluation by inspection and a brief MOS survey, we find that our proposed framework results in a significant improvement over the museum's model, especially in terms of facial resemblance. We also demonstrate that the framework could be implemented at scale as one-step solution in a commercial setting. As such, we recommend that the museum implement this methodology in lieu of their model in order to effectively and equitably engage with visitors of all demographics.