People we despise, sure. But not people who break the social contract. Using their freedom of expression to restrict the freedoms of others, or to advocate for the same. That behaviour violates the contract, and anyone who violates the contract is not covered by it.