That’s an orthogonal thing though. Downvotes and removals are generally triggered by different things, even if they intersect occasionally:

• “I think The Mandalorian is the best sci-fi western ever” is a reasonable thing to post in c/tv, c/scifi, c/westerns, etc.
• “I think The Mandalorian is the best sci-fi western ever” might be a reasonable thing in an open topic about comparing Firefly to other space westerns in c/firefly
• "I think The Mandalorian is the best sci-fi western ever" is clearly an off topic troll if posted as a top level reply to a question “Do you think Mal will ever find his true love” in c/firefly

In all three cases, it’s entirely possible the comment will get a lot of downvotes. But the fact it’s off-topic and, in context, intended to troll, that means the third case is the only one where you’d want it removed.

Indeed, it’s possible to envisage a highly upvoted comment that also ends up being removed because it’s off topic or an attempt to derail. Those are actually harder to remove, and I’ve seen (Reddit) moderators make the wrong call on them when they’re successful attempts to derail because they’re not always as obvious.