Assuming the training software could be run on the hardware and that we could distribute the load as if it was 2023, would it be possible to train a modern LLM on hardware from 1985?

  • ResponsibleJudge3172@alien.topB
    link
    fedilink
    English
    arrow-up
    1
    ·
    1 year ago

    No, you are limited by:

    Compute Performance, you will need 10,000%+ more compute than was available per chip, and those PCIe accelerators don’t have the ability to compute the way they do now. You are going to have to rely on CPUs which is worse

    Lack of scalabality of interconnecting chips to behave as one, increasing IO requirements dramatically.

    Lack of memory pooling (yes you qualified it), memory bandwidth and memory sizes (we are talking in megabytes), imagine waiting for 1 billion parameter model calculations to load and store in each layer of a neural network using floppy disks.

  • Ronny_Jotten@alien.topB
    link
    fedilink
    English
    arrow-up
    1
    ·
    1 year ago

    Do you mean one computer from 1985? No. There is no computer from that year that had enough RAM. If you mean all the computers from 1985, working together, then yes. You only need sufficient RAM, a Turing-complete machine, and probably some centuries to do it.

    • stinkypeteryerg@alien.topOPB
      link
      fedilink
      English
      arrow-up
      1
      ·
      1 year ago

      Yeah - you got it right. I’m looking to see if it was hyoothetically feasible, not whether it was practical. I know it wouldn’t be practical

  • DannyBoy
    link
    fedilink
    English
    arrow-up
    1
    ·
    1 year ago

    The fastest computer in 1985 was the CRAY-2 supercomputer at 1.9 gigaflops. ChatGPT 3 can be trained on 1024 A100 GPUs in 34 days*. An A100 outputs 312 teraflops. So no, I don’t think it can be done in 1985 if given the entire year. There’s also storage for incoming digital texts for training - the input data didn’t exist back then, not to the capacity. I don’t think it could be done in a reasonable time.

  • StrivingShadow@alien.topB
    link
    fedilink
    English
    arrow-up
    2
    arrow-down
    1
    ·
    1 year ago

    It wouldn’t. Training the neural nets for LLMs are all about brute force, and it’s only been possible in the last few years to train these models without spending billions. Even going back to 2010 I think it’d be largely infeasible.

    The good news is if we fast forward even just a few years, training will become relatively cheap compared to today.

    • Ronny_Jotten@alien.topB
      link
      fedilink
      English
      arrow-up
      1
      ·
      1 year ago

      They didn’t ask if it it could be done without spending billions, or whether it would be feasible, i.e., practical, just whether it would be possible.

    • tomytronics@alien.topB
      link
      fedilink
      English
      arrow-up
      2
      arrow-down
      1
      ·
      1 year ago

      So my collection of about 20 Commodore 64 isn’t enough? Do I need another 20? /s

  • SkitzMon@alien.topB
    link
    fedilink
    English
    arrow-up
    1
    ·
    1 year ago

    The growth rate in computing has been exponential. Using the fastest available computer from 1985 continuously for 38 years and still going, you would be passed by a quad GPU-based server in hours.