I’d also argue that by removing it (especially at this time) you’re in fact doing the opposite—you’re now inciting racism because the racists will call it a win and take it as a sign that they’ve “won” or it’s acceptable now.