As others have already said, exposure can be a good way to overcome fear. Start by going somewhere nearby. Maybe ask a friend or someone you feel safe with to accompany you a few times until you are familiar with the route or area. Download an offline map in case you get lost. 

You said you feel out of place at your local library , but they’re a quiet, safe place where everyone generally minds their own business. You can probably even bring take your own ebooks there to read. You don’t have to interact with anyone if you don’t want to, just get used to being in a different setting.

Can you take driving lessons? That could be another way to familiarize yourself with different areas and find more independence.