Well, you got up. Point made. No reason to fear anymore.

I would not exactly answer to what he said, but say a different thing, maybe like “You want to sit here?” (polite on the surface, stating the obvious, but implicitly putting the focus on on his action, not yours)