It means that the person you are talking to believes that you’re just making polite conversation because you’re uncomfortable with silence. They are letting you know that they would prefer you didn’t make conversation, because they prefer silence to idle chatter.