Do i need industry grade gpu’s or can i scrape by getring decent tps with a consumer level gpu.

  • ffhein@lemmy.world
    link
    fedilink
    English
    arrow-up
    2
    ·
    1 day ago

    You have to specify which quantization you find acceptable, and which context size you require. I think the most affordable option to run large models locally is still getting multiple RTX3090 cards, and I guess you probably need 3 or 4 of those depending on quantization and context.