The rep points are messed up on both platforms.
I guess when they decide an account has X number of downvotes that they restrict our access, we can worry about it. But right now, up/down has no real meaning.
Even on reddit I remember people getting mass downvotes and nothing ever happened to those accounts.
Take every down vote as a way to remind yourself that "someone is thinking about me". :)