It isn’t a big deal, but we do need the language to evolve a little bit. The problem with they/them is that it implies that you don’t know the person, or that it doesn’t matter who they are (like you say, you can’t or don’t want to use a more specific pronoun). It can feel quite rude to apply it to somebody that you do know.