Transcript-Driven AI Video Editing: Streamlining Production with Natural Language Processing and Computer Vision

Abstract

Video has become the dominant form of online communication, yet video editing remains a tedious process. One of the most time-consuming tasks for editors is analyzing and cutting down long footage in talking videos like interviews, podcasts and commentary content. This paper introduces STREAM, a Smart Transcript Rendering and Analysis Model that leverages state-of-the-art Natural Language Processing and Computer Vision techniques to streamline the video editing workflow. By "seeing," "hearing," and "reasoning" like a human editor, STREAM empowers users by transforming high-level editing instructions into polished videos. The system employs extractive summarization to highlight significant clips from the video, Large Language Models to perform sentiment analysis and select appropriate effects, and Computer Vision to create speaker-focused scenes. A human-in-the-loop approach ensures accuracy while a range of customization options cater to diverse editing styles and needs. Studies with users of varying video editing experience levels demonstrate STREAM's potential to optimize and democratize the craft of video editing. STREAM achieves an average 77% reduction in transcription time compared to traditional subtitling methods. Ultimately, STREAM aims not to replace human editors but rather augment their creative potential in an AI-assisted future.