Main link is to GPU image, CPU image can be found here:
https://hub.docker.com/r/noneabove1182/text-gen-ui-cpu
The CPU one is built for exclusively running with a CPU. The GPU one is compiled with CUDA support and gets blazing fast ingestion and generation.
Included in each readme is a disclaimer that I am once again not affiliated, and I include an example working docker-compose.yml, make sure you change the args to fit your own personal setup! :)
Feel free to ask any questions or let me know if anything doesn’t work! Hacked it together by the skin of my teeth, and put a LOT of effort into reducing image size for the GPU one (16GB down to 9GB, still massive…) so please do post if you have any issues!
I’m also working on getting koboldcpp working with GPU support, currently it works for ingestion but for some reason the generation itself is still pretty slow… will post an update when it’s working well!