Popper’s paradox of tolerance gives in my view pretty clear guidelines on what to protect and what not to tolerate. I believe that if we held onto that, fascism would have a much harder time.

Popper did make his line clear: physical violence.

“I do not imply, for instance, that we should always suppress the utterance of intolerant philosophies; as long as we can counter them by rational argument and keep them in check by public opinion, suppression would certainly be most unwise But we should claim the right to suppress them if necessary even by force; for it may easily turn out that they are not prepared to meet us on the level of rational argument, but begin by denouncing all argument they may forbid their followers to listen to rational argument, because it is deceptive, and teach them to answer arguments by the use of their fists or pistols.”

So I agree with you. Tolerate up to the point of people using physical violence to enact their political aims.