You’re describing the Fallacy Fallacy, being that the implication that the argument is necessarily wrong because a fallacy has been committed. That a fallacy has been committed by the other party should not alone be used as an argument against the point itself.

I.e. you committed a strawman fallacy by stating that all strawmen are made of straw, therefore no strawmen are made of straw