My current PC specs:

  • Dell Precision T1700 Motherboard
  • Xeon E3 1275 v3
  • 32GB DDR3 1600Mhz ECC
  • 7900 XTX 24GB VRAM (Transferring to new PC)

What I want in my next PC:

  • MSI Z790-P Motherboard (with Dasharo)
  • i9-14900k
  • DDR5 192GB 5600Mhz (Non-ECC)
  • 7900 XTX 24GB VRAM

Will I see any performance increase? I’m also worried about not having ECC RAM, I will be coding a lot with my LLMs (firmware development), so part of me doesn’t want to make any mistakes and future proof. I heard DDR5 has some ECC-like features, but not the same as traditional ECC. Maybe I should buy a Dell Precision 3630 motherboard which has support for upto 128GB DDR4 ECC memory? This is a dedicated LLM PC, so, not too concerned about gaming or anything.

I would like to hear your suggestions!

  • april
    link
    fedilink
    English
    476 days ago

    Only the GPU and primarily the vram matters for LLMs. So this wouldn’t help at all.

    • mozz
      link
      fedilink
      10
      edit-2
      6 days ago

      You’re the only one talking sense and you are sitting here with your 2 upvotes

      The AI company business model is 100% unsustainable. It’s hard to say when they will get sick of hemorrhaging money by giving away this stuff more or less for free, but it might be soon. That’s totally separate from any legal issues that might come up. If you care about this stuff, learning about doing it locally and having a self hosted solution in place might not be a bad idea.

      But upgrading anything aside from your GPU+VRAM is a pure and unfettered waste of money in that endeavor.

    • @[email protected]
      link
      fedilink
      English
      3
      edit-2
      6 days ago

      GPU with a ton of vran is what you need, BUT

      An alternate solution is something like a Mac mini with an m series chip and 16gb of unified memory. The neural cores on apple silicon are actually pretty impressive and since they use unified memory the models would have access to whatever the system has.

      I only mention it because a Mac mini might be cheaper than GPU with tons of vram by a couple hundred bucks.

      And it will sip power comparatively.

      4090 with 24gb of vram is $1900 M2 Mac mini with 24gb is $1000

      • @[email protected]
        link
        fedilink
        English
        36 days ago

        Buying second hand 3090/7090xtx will be cheaper for better performances if you are not building the rest of the machine.

    • TimeOP
      link
      English
      2
      edit-2
      5 days ago

      Don’t you need tons of RAM to run LLMs? I thought the newer models needed up to 64GB RAM? Also, what about Stable Diffusion?

      • @atzanteol
        link
        English
        126 days ago

        VRAM. Not system RAM. LLMs run best entirely on the GPU.

      • april
        link
        fedilink
        English
        8
        edit-2
        6 days ago

        Ram is important but it has to be vram not system ram.

        Only MacBooks can use the system ram because they have an integrated GPU rather than a dedicated one.

        Stable diffusion is the same situation.

      • Pumpkin Escobar
        link
        fedilink
        English
        66 days ago

        Taking ollama for instance, either the whole model runs in vram and compute is done on the gpu, or it runs in system ram and compute is done on the cpu. Running models on CPU is horribly slow. You won’t want to do it for large models

        LM studio and others allow you to run part of the model on GPU and part on CPU, splitting memory requirements but still pretty slow.

        Even the smaller 7B parameter models run pretty slow in CPU and the huge models are orders of magnitude slower

        So technically more system ram will let you run some larger models but you will quickly figure out you just don’t want to do it.

      • @[email protected]
        link
        fedilink
        English
        25 days ago

        They do, but VRAM. Unfortunately, the cards that do have that much of memory are used by OEMs/corporations and are insanely pricey

    • enkers
      link
      English
      1
      edit-2
      6 days ago

      One minor caveat where CPU could matter is AVX support. I couldn’t get ollama to run well on my system, despite having a decent GPU, because I’m using an ancient processor.

  • @[email protected]
    link
    fedilink
    English
    35 days ago

    I9 14900k…bad news for you, 13th and 14th gen I9 is unstable, crashes.

    Suggestion: Wait for 15th gen or AMD 9000 series CPU to come out.

  • adr1an
    link
    fedilink
    English
    15 days ago

    Have you tried ollama ? Some (if not all) models would do inference just fine with your current specs. Of course, it all depends on how many queries per unit of time you need. And if you wanted to load a huge codebase and pass it as input. Anyway, go try out.

  • @[email protected]
    link
    fedilink
    English
    26 days ago

    Will I see any performance increase?

    Like others have said LLMs mostly use VRAM, they can use system RAM if you’re running them on CPU, but that’s ridiculously slow.

    It will however increase the speed of your compile times, which is especially useful if you’re compiling something large like the Linux kernel on a regular basis.

    I’m also worried about not having ECC RAM.

    If you are using it purely for LLMs, if it’s going to get bit flips, it’ll happen in VRAM.

    If you are compiling large things for customers, I’d recommend ECC, just in case, e.g. you don’t want a bricking firmware from a bit flip. But according to EDAC and my TIG stack, my server’s ECC RAM has never even detected an error in the past year, if I understand EDAC properly, so it’s really not important.

  • @[email protected]
    link
    fedilink
    English
    16 days ago

    Have you looked into specialized AI chips/accelerators at all if you really want to mess with it?

    Way lower end than what you’re working with, but they have AI accelerator kits for something as small as a Raspberry Pi.

    • @[email protected]
      link
      fedilink
      English
      26 days ago

      You are limited by bandwidth not compute with llm, so accelerator won’t change the interferance tp/s