I think of it exactly in terms of the trolley problem. The whole premise is that if you do nothing (don’t vote) more people die. By flipping the lever, fewer people die but you’ve taken an action that leads directly to their deaths. The philosophical question isn’t just “is it better for fewer people to die” but “in pulling the lever, are you directly responsible for those deaths?”

My answer would be that inaction is itself an action. In this scenario, you have found yourself responsible either way. Suppose you pull the lever, though, to save as many lives as you can… Wouldn’t the ones who die as a result of this have loved ones that absolutely do blame you?