Since I blocked Lemmy.ml, I haven't downvoted anything at all for several weeks, which is surprising especially given the USA election. Lemmy's userbase really is 1000-fold more enjoyable to interact with than e.g. Reddit.

Overall I only downvote someone acting as a troll, and often even then just ignore it and move on.