Depends upon what you want to be safe from.

If you want to be safe from having a flash mob come and beat you up, I’d say, this assumption doesn’t really compromise your safety.
If you want to be safe from having the wrong assumption, well, you are deep in unsafe territory.