This is a new family of open source video models designed to take on Gen-2. There is a 576x320 model that uses under 8gb of vram, and the 1024x576 model that uses under 16gb of vram. The recommended workflow is to render with the 576 model, then use vid2vid via the 1111 text2video extension to upscale to 1024x576. This allows for better compositions overall and faster exploration of ideas before committing to a high res render.
This is currently the best text2video model, that i have seen. But it has some problems with objects morphing.
You can try it out here: https://replicate.com/anotherjesse/zeroscope-v2-xl
Just tried it and the result was goofy but the frame consistency and smoothness of movement is insane! I can’t wait to see how things develop.
The speed of progress that we are making with ai is crazyyy. I will be able to watch quality movies generated by ai sooner than I was expecting. I thought it will take 5 years or more at best case scenario but now I think that it will be less than 5 years from now.
At the current rate of progress, completely generated films are probably possible next year. The audio and video part is currently not good enough, but the quality will probably get to Midjourney5 level in the next half year. Scripts for a full movie can be written by GPT-4, currently it still needs a lot of help for a good result, but with better fine tuning that shouldn’t be a problem. Then the audio and video parts can be combined by using ChatGPT code interpreter, which already works quite well.