ToonCrafter Overview: Consistent Cartoon Interpolation Like Never Seen Before!

Making animations using Generative AI has become even easier with ToonCrafter, a new technique for creating consistent videos from input frames.

The core feature of ToonCrafter is its ability to consistently fill the ‘gaps’ between two or more input frames, producing natural and smooth cartoon animation. In other words, you provide a starting image and a final image; ToonCrafter will create the content between these two.

This is an area where many generative AI tools currently struggle (such as Stable Video Diffusion, which lacks control and produces artifacts) or require several gimmicks (like AnimateDiff with ControlNet) to get good results.

You still need to provide consistent input frames when using ToonCrafter, as this tool is designed for interpolation (merging still images). However, it’s much easier to create or draw two consistent images rather than every single frame needed for a full animation.

ToonCrafter is not the first of its kind; there are other tools like AnimeInterp, SEINE, and EISAI. However, the strength of ToonCrafter lies in its ability to produce smooth and consistent animations with minimal artifacts or distortions. It really knows how to fill what’s missing!

It can be used to interpolate black-and-white cartoon sketches, as well as to colorize animations using an input reference, or interpolate between two colored input images to serve as the start and end of the animation.

In general, ToonCrafter can support generating videos of up to 16 frames with a resolution of 512x320. The inference time can be reduced by using fewer DDIM steps; however, the GPU memory requirements are rather high. But optimizations are always just around the corner!

Want to experiment with ToonCrafter? Choose your input images and try it out on this page. The original models can also be downloaded from Doubiiu/tooncrafter, but these are not yet optimized and come in .ckpt format.

As we know, the community is fast, and a ComfyUI wrapper is already available named DynamiCrafter. This optimized custom node can generate 256x256 animations with just 8GB of RAM. However, for the full ToonCrafter resolution of 512x320, you’ll need 15GB for sampling, with a peak of around 17GB during decoding.

To get all the benefit of this wrapper, download also the optimized models in safetensor fromat form Kijai/DynamiCrafter_pruned.

ToonCrafter seems to be one of the most promising tools for generative animations that we can all download and use locally. SORA from OpenAI was impressive, but no one can try it yet. Other closed-source solutions like Pikalabs and RunwayML haven’t reached this level of consistency yet, though they will probably integrate this functionality soon.

Looking forward to experimenting with ToonCrafter and seeing the improvements that come from now on!