Could someone recommend a LLM for the Nvidia GTX1080? I’ve used the gptq_model-4bit-128g of Luna AI from the Bloke and i get a response every 30s-60s and only 4-5 prompts before it starts to repeat or hallucinate.

    • pythia@lemmy.dbzer0.comOP
      link
      fedilink
      English
      arrow-up
      2
      ·
      1 year ago

      Yes it does and fits the GPU just fine. Didn’t hallucinate but it was slow like 60s+ in the first run but did it’s job. Thanks.

      • SkySyrup
        link
        fedilink
        English
        arrow-up
        2
        ·
        1 year ago

        good to hear it worked, it’s weird it’s so slow. I’m lucky to have access to a 3060, which isn’t that far out from a 1080, and get at least 40t/s on it. Are you running on CPU or are you using exllama?

          • SkySyrup
            link
            fedilink
            English
            arrow-up
            2
            ·
            1 year ago

            that’s really weird, I’m not sure how to help you there unfortunately :(