It isn’t the only instance where it does that; it also happens in several other places like most shops and minigame houses, the barn at Lon Lon Ranch, Dampe’s shack and most house interiors, including Link’s House, the very first location in the game. Castle Town is the largest and most prominent, and you easily spend the most time there, so you notice it there more. Some of the locations I mentioned aren’t required and you might completely miss them, but not Castle Town.