That’s like asserting that a self-defense claim is an argument that you didn’t hit the other guy. You really did hit him (copyright infringement / assault), but you have a defense that admits the literal facts but absolves you of liability (fair use / self-defense.)

You don’t need to argue self-defense if you can convince the court that you didn’t actually hit the other guy.