Text-to-Video Generation Based on Diffusion Model

Abstract

Text-to-video generation is a fundamental yet challenging task in generative AI, requiring models to synthesize temporally coherent and visually compelling videos from textual descriptions. It has broad applications in content creation, storytelling, virtual reality, and assistive technologies. However, ensuring realism, semantic alignment, and controllability in generated videos while maintaining computational efficiency remains a significant challenge.

Diffusion models, which have demonstrated state-of-the-art performance in image generation, offer a promising framework for video generation. Their iterative denoising process enables stable training, high-fidelity synthesis, and effective multimodal conditioning. Extending diffusion models to video generation introduces new challenges, such as maintaining temporal coherence and aligning motion with textual inputs.

This thesis explores text-to-video generation with diffusion models across three key areas: training and sampling methodology, foundational diffusion model improvements, and evaluation. First, we develop novel approaches for text-to-video diffusion model training and sampling, addressing challenges such as learning text-to-video generation without paired datasets, parameter-efficient text-to-video diffusion model, controllable camera motion, and long-range video generation. Second, we investigate ways to enhance diffusion models by integrating metric functions to improve sample fidelity and diversity. Third, we introduce new evaluation methods, leveraging multimodal LLMs to assess interleaved image-text generation and proposing techniques to quantify the artistic quality of images and videos.

By tackling these challenges, this thesis advances the capabilities of diffusion-based text-to-video generation, providing both insights and practical innovations. Finally, we discuss promising directions for future research.