You’re fine, he may have been mentally challenged, he may be immersed in a mall walking social group that has formed there and thought it was a risk worth taking based on previously successful or milder experiences, but you are well within your rights socially and everything else to be firm and curt and avoidant. I applaud your continued assessment and concern, it shows you actually care about people’s feelings, so imo you’re fine and will do great with whatever options you go with in the future. If you want my suggestion for a slightly easier refusal, I’d maybe just say “I really do appreciate your being social and being welcoming, I’m just not up for socializing today, sorry” and that should do it. But I think you did fine.