Writing a 100-word email using ChatGPT (GPT-4, latest model) consumes 1 x 500ml bottle of water It uses 140Wh of energy, enough for 7 full charges of an iPhone Pro Max
Writing a 100-word email using ChatGPT (GPT-4, latest model) consumes 1 x 500ml bottle of water It uses 140Wh of energy, enough for 7 full charges of an iPhone Pro Max
I’ve run an LLM on my desktop GPU and gotten decent results, albeit not nearly as good as what ChatGPT will get you.
Probably used less than 0.1Wh per response.
Is this for inferencing only? Do you include training?
Training is a one time thing. Tge more it get use, the less energy per query it will take
Good point. But considering the frequent retraining, the environmental impacts can only be spread on a finite number of queries.
They have already reached diminishing returns on training. It will become much less frequent soon. Retraining on the same data if there isn’t a better method is useless. I think the ressources consumed per query should only include those actually used for inference. The rest can be dismissed as bad faith argumentation.
Inference only. I’m looking into doing some fine tuning. Training from scratch is another story.