Raise your hand if you feel like an oppressor.
...Did you raise your hand?
When you take a moral stance, on some level you're accusing others of taking an immoral stance. When you fight for justice, it must mean some people are unjust. There's not really a way around that unless you draw really vague boundaries.