I think there is an option in Settings. Under privacy there is a toggle for Listening Activity to hide it from followers. It’s below the temporary version of the setting, Private Session I think you are thinking of. Why a temporary privacy setting would be such an option is beyond me.