Just to preface, I’m a scientist: micro- and molecular biology. I’m not saying to take what I say as gospel, just giving context that I might know things. Sometimes.

Outbreeding depression has more possible implications than fertility decrease and infant mortality increase, entirely dependent on the heritable traits responsible for the depression effects. While the probability of persistent outbreeding depression as a result of one or more crossings would be lower in traits subject to higher selective pressure, such as increases in early infant mortality, the overall probability of outbreeding depression itself isn’t influenced post facto by its results, just its persistence.

Given we don’t know the original extent of neanderthal/human interbreeding, what we’re seeing now COULD be the “much lower percentage” you mention and still could come from multiple events. In fact, if these crosses resulted in stronger depression effects, I’d argue a greater number of crossings would be one factor behind the persistence of some genes today.