Assuming the training software could be run on the hardware and that we could distribute the load as if it was 2023, would it be possible to train a modern LLM on hardware from 1985?
Assuming the training software could be run on the hardware and that we could distribute the load as if it was 2023, would it be possible to train a modern LLM on hardware from 1985?
No, you are limited by:
Compute Performance, you will need 10,000%+ more compute than was available per chip, and those PCIe accelerators don’t have the ability to compute the way they do now. You are going to have to rely on CPUs which is worse
Lack of scalabality of interconnecting chips to behave as one, increasing IO requirements dramatically.
Lack of memory pooling (yes you qualified it), memory bandwidth and memory sizes (we are talking in megabytes), imagine waiting for 1 billion parameter model calculations to load and store in each layer of a neural network using floppy disks.