“The enormous costs” were optional, and driven by greed. This latest neural-network revolution happened thanks to consumer video game hardware. Then the big boys got involved and pushed unreasonably large models, partially because they figured that’s where the magic would happen… but partially to justify all the data they’d need to scoop up indiscriminately… and partially because going big limited competition.
Google themselves proved this was not necessary. Remember the AI that could beat any human at Go? Well, they built a smaller model as a blank slate, and just had it play games, and it quickly got better than the big news-worthy AI. Then they built an even smaller model, and made it do a bunch of games instead of just Go, and it still got better than either previous AI, at Go.
“The enormous costs” were optional, and driven by greed. This latest neural-network revolution happened thanks to consumer video game hardware. Then the big boys got involved and pushed unreasonably large models, partially because they figured that’s where the magic would happen… but partially to justify all the data they’d need to scoop up indiscriminately… and partially because going big limited competition.
Google themselves proved this was not necessary. Remember the AI that could beat any human at Go? Well, they built a smaller model as a blank slate, and just had it play games, and it quickly got better than the big news-worthy AI. Then they built an even smaller model, and made it do a bunch of games instead of just Go, and it still got better than either previous AI, at Go.