So what is currently the best and easiest way to use an AMD GPU for reference I own a rx6700xt and wanted to run 13B model maybe superhot but I’m not sure if my vram is enough for that Since now I always sticked with llamacpp since it’s quiet easy to setup Does anyone have any suggestion?

    • actually-a-catEnglish
      3·
      1 year ago

      Not sure what happened to this comment… Anyway, ooba (text-generation-webui) works with AMD on Linux but ROCm is super jank at the best of times and 6700XT is not officially supported so it might be hopeless.

      llama.cpp has some GPU acceleration support on AMD in CLBlast mode, if you aren’t already using it, might be worth trying.

      • Mixel@feddit.deOPEnglish
        2·
        1 year ago

        How do use ooba with rocm I looked at the python file where you can install amd and it will just say “amd not supported” and exit. I guess it just doesn’t update the webui.py when I update ooba? I somewhere heard that llama.cpp with CLBlast wouldn’t work with ooba, or am I wrong? Also is konoldcpp worth a shot? I hear some success with it

        • actually-a-catEnglish
          5·
          1 year ago

          I can recommend kobold, it’s a lot simpler to set up than ooba and usually runs faster too.

          • Mixel@feddit.deOPEnglish
            3·
            1 year ago

            I will try that once in home! Ty for the suggestions can I use kobold also in sillytavern? iirc there was an option for koboldai or something is that koboldcpp or what does that option do?

            EDIT: I got it working and its wonderful thank you for suggesting me this :) I had some difficulties setting it up especially with opencl-mesa since I had to install opencl-amd and then finind out the device ID and so on but once it was working its great!