You absolutely can go have lunch elsewhere. I’ve been in similar situation. If asked, you can simply tell them, I enjoy having lunch by myself, it helps me recharge. Also, most of the time, boundaries are set through action not only words. Just do what you prefer without the concern of what others will think or feel, while being polite with your words. Most people will pick up on you actions and eventually leave up be. I’ve had serious boundary issues in my family and I’ve had to learn quite a bit about forming proper sustainable boundaries.