Yes. Don’t put people in uncomfortable situations.

But also, if they don’t know about the real cause, you may still hurt their feelings. If you can find another reason to decline it may go better.