I’d consider saying that rude most of the time as the person making small talk is just doing something nice.

the question here is how many hints/suggestions/requests has OP already blown past? if all they talk about is the weather, chances are you should just let the conversation drop.