There's a fairly 101 level anti-racist idea which is "intent is not impact". Just because you didn't _mean_ to be racist does not mean that your action didn't come across as a microaggression (or worse). I feel like your point about wearing masks is a nice corollary to that.

It literally doesn't matter what your intent is for wearing a mask! Whether you're doing it to protect yourself, to protect others, to look really cool, or to signal virtue the effect is going to be the same.