I think it's good you listened to your instincts and were firm. If he has a disability that makes social things complicated, being obvious was a favor to him. If he did have bad intentions, your immediate willingness to leave showed you weren't an easy mark.

There are probably good scripts for this, but I don't know them. I just think you should know it doesn't sound like you handled it badly.