Easy Text to Video with AnimateDiff

AnimateDiff lets you easily create videos using Stable Diffusion. Just write a prompt, select a model, and activate AnimateDiff!

what is AnimateDiff

AnimateDiff is an AI tool that can turn a static image or text prompt into an animated video by generating a sequence of images that transition smoothly. It works by utilizing Stable Diffusion models along with separate motion modules to predict the motion between frames. AnimateDiff allows users to easily create short animated clips without needing to manually create each frame.

Key Features of AnimateDiff

AnimateDiff can generate animations from text prompts alone.

Users can upload an image and AnimateDiff will predict motion to generate an animation.

Users don't need to manually create each frame, as AnimateDiff automatically generates the image sequence.

AnimateDiff can be seamlessly integrated with Stable Diffusion and leverage its powerful image generation capabilities.

How does AnimateDiff work

  • It utilizes a pretrained motion module along with a Stable Diffusion image generation model.

  • The motion module is trained on a diverse set of short video clips to learn common motions and transitions.

  • When generating a video, the motion module takes a text prompt and preceding frames as input.

  • It then predicts the motion and scene dynamics to transition between frames smoothly.

  • These motion predictions are passed to Stable Diffusion to generate the actual image content in each frame.

  • Stable Diffusion creates images that match the text prompt while conforming to the motion predicted by the module.

  • This coordinated process results in a sequence of images that form a smooth, high-quality animation from the text description.

  • By leveraging both motion prediction and image synthesis, AnimateDiff automates animated video generation.

What are some potential use cases and applications for AnimateDiff

Art and animation

Artists/animators can quickly prototype animations and animated sketches from text prompts. Saves significant manual effort.

Concept visualization

Helps visualize abstract concepts and ideas by turning them into animations. Useful for storyboarding.

Game development

Can rapidly generate character motions and animations for prototyping game mechanics and interactions.

Motion graphics

Create dynamic motion graphics for videos, ads, presentations etc. in a highly automated way.

Augmented reality

Animate AR characters and objects by generating smoother and more natural motions.


Preview complex scenes with animation before filming or rendering final production.


Create explanations and demonstrations of concepts as engaging animated videos.

Social media

Generate catchy animated posts and stories by simply describing them in text.

The capability to go directly from text/images to animation opens up many possibilities for easier and more rapid animation creation across several domains.

How to use AnimateDiff

You can use AnimateDiff for free on the animatediff.org website without needing your own computing resources or coding knowledge. On the site, you simply enter a text prompt describing the animation you want to create. AnimateDiff will then automatically generate a short animated GIF from your text prompt using state-of-the-art AI capabilities. The whole process happens online and you can download the resulting animation to use as you like. This provides an easy way to experience Animatediff's animation powers without setup. You can start creating AI-powered animations from your imagination in just a few clicks!

What are the system requirements for running AnimateDiff

An Nvidia GPU is required, ideally with at least 8GB VRAM for text-to-video generation. 10+ GB VRAM needed for video-to-video.

A sufficiently powerful GPU for inference is needed, like an RTX 3060 or better. The more powerful the GPU, the better the performance.

Windows or Linux. macOS can work through Docker. Google Colab is also an option.

16GB system RAM minimum recommended.

A decent amount of storage is required for saving image sequences, videos, and model files. At least 1 TB is recommended.

Works with AUTOMATIC1111 or Google Colab. Requires installing Python and other dependencies.

Currently only compatible with Stable Diffusion v1.5 models.

Overall, AnimateDiff works best with a powerful Nvidia GPU with abundant VRAM and compute capability, running on Windows or Linux. Lacking a strong enough GPU can result in slow generation speeds or issues with video quality.

How to Install the AnimateDiff extension

  • Start the AUTOMATIC1111 Web UI normally.

  • Go to the Extensions page and click on the "Install from URL" tab.

  • In the URL field, enter the Github URL for the AnimateDiff extension: https://github.com/continue-revolution/sd-webui-animatediff

  • Wait for the confirmation that the installation is complete.

  • Restart the AUTOMATIC1111 Web UI.

  • The AnimateDiff extension should now be installed and visible in the txt2img and img2img tabs.

  • Download the required motion modules and place them in the proper folders as explained in the documentation.

  • Restart AUTOMATIC1111 again after adding motion modules.

Now the AnimateDiff extension is installed and ready to use for generating animated videos in AUTOMATIC1111!

Advanced options about AnimateDiff

Close loop

Makes the first and last frames identical to create a seamless looping video.

Reverse frames

Doubles the video length by appending frames in reverse order. Creates more fluid transitions.

Frame interpolation

Increases frame rate to make motion look smoother.

Context batch size

Controls temporal consistency between frames. Higher values make changes more gradual.

Motion LoRA

Adds camera motion effects like panning, zooming, etc. Works similarly to regular LoRA.


Directs motion based on a reference video's motions using ControlNet capabilities.


Allows defining start and end frames to have more control over composition.


Frames per second control the speed of the animation.

Number of frames

Determines the total length of the generated video.

Motion modules

Different modules produce different motion effects.

By tweaking these settings, one can achieve more control over the style, smoothness, camera motion, speed, and length of the AnimateDiff videos.

What are some current limitations of AnimateDiff

Limited motion range

The motions are constrained by what's in the training data. It cannot animate very complex or unusual motions not seen in the training set.

Generic movements

The motion is not tailored specifically to the prompt, so it tends to produce generic movements loosely related to the prompt.


Can sometimes produce visual artifacts as motion increases.


Currently only works with Stable Diffusion v1.5 models. Not compatible with SD v2.0.

Training data dependence

Quality of motion relies heavily on diversity and relevance of training data.

Hyper parameter tuning

Getting smooth, high-quality motion requires tuning many settings like batch size, FPS, frames, etc.

Motion coherence

Maintaining logical motion coherence over long videos is still a challenge.

While-capable of generating short, basic animations from text, AnimateDiff has limitations around complex motions, motion quality, and seamless transitions. As the technology matures, we can expect many of these issues to be addressed.