‘I’m not really here to talk’ and leaving isn’t rude at all, don’t sweat it. If you had screamed at him that he was a creepy POS or something then you’d have a good reason to feel bad if he had some kind of intellectual/developmental disability, but you didn’t, and even if he does there’s a point where you have to learn you’re not a cute kid who can just walk up to people at random and start talking to them anymore (or at least not have everyone be as receptive to it).