It seems people missed the “with wives” part. Generally speaking, your wife is not going to be terribly pleased with you ogling or getting off to other women. And violating that is what makes these men count as weird.