Your first sentence says, “it actually was that bad,” but your second sentence says, “but it wasn’t that bad” so I’m going to assume you meant the latter.