Could someone recommend a LLM for the Nvidia GTX1080? I’ve used the gptq_model-4bit-128g of Luna AI from the Bloke and i get a response every 30s-60s and only 4-5 prompts before it starts to repeat or hallucinate.
Could someone recommend a LLM for the Nvidia GTX1080? I’ve used the gptq_model-4bit-128g of Luna AI from the Bloke and i get a response every 30s-60s and only 4-5 prompts before it starts to repeat or hallucinate.
Yes it does and fits the GPU just fine. Didn’t hallucinate but it was slow like 60s+ in the first run but did it’s job. Thanks.
good to hear it worked, it’s weird it’s so slow. I’m lucky to have access to a 3060, which isn’t that far out from a 1080, and get at least 40t/s on it. Are you running on CPU or are you using exllama?
It’s running on gpu, the task-manager shows 92% GPU utilization and i chose exllamaV2.
that’s really weird, I’m not sure how to help you there unfortunately :(