Hmm you’re right. But the hours-long duel Dumbledore had with Grindelwald is mentioned as something only he could do. So it might be that magical energy is limited, but you’d need to be casting spells for more than an hour or so for it to start having an effect.