Up and down votes are intended as indicators of quality that people abuse as markers of taste. If you downvote because the post/comment is unhelpful/useless/etc. that’s just voting as intended. If you are about to downvote because something is hateful, just report it to the mods instead. If you use certain clients, like Voyager, you will also see cumulative totals for your personal votes on users, which can be a great way to spot people you shouldn’t interact with, and probably just block. If you can’t tell the difference between people who are willfully nasty vs ignorant, maybe focus on learning to tell that rather than participate too much.