You don’t owe anyone an explanation. You can just say that you’re not coming and leave it at that. You can tell them that you have your reasons and you wish not to talk about it but don’t lie. That just makes you the asshole.