You might just need to be more honest and direct. Something like “thanks for the invite but i want to go solo today”