I have several family members who either don’t drive at all or have previously taken several-year-long breaks from driving.

I was significantly older than you are now when I learned to drive at all, I no longer own a car and haven’t driven in approximately 9 months.

If you are in a life situation where you do not need to drive, then it is completely your own decision and you shouldn’t listen to anyone else trying to convince you one way or the other. Most drivers overestimate their abilities; if you yourself feel driving isn’t for you, then it’s better for everyone involved (including others on the road) that you don’t drive. But you still have a long life ahead of you and it’s completely possible you’ll become more confident when some more years have passed.