‘No.’ is a complete sentence and you do not have to justify yourself in any way. I don’t think that would be considered rude, either.