If 999/1000 people think a green flag with a purple unicorn is a symbol of racism, then it's a symbol of racism. No matter how accurate that flag is or what it's original intentions were it's a symbol Of racism. As such it shouldn't be flying over any municipal building.
That said you want to put it in your front yard be my guest, it's your right to do so even if it makes you a complete asshole.