I generally shy away from downvoting even things I dislike. I don’t want to fall into an echo chamber where all I see are only the things I agree with.

This is not how Lemmy works. If you don’t ever up or down vote anything, and if you have your client hide the up/down votes on all comments and posts, you will still see the same content as someone who prodigiously votes. Votes here are, in the truest possible way, optional.

But for people who do utilize up/down votes, they are a signal. Sometimes they’re a weak signal, sometimes a bad signal, and yet other times might be a strong signal from the community. How to evaluate the signal is a matter of continued debate.