This actually stems from long forgotten common courtesy to remove a hat when entering someone’s home or place of business.

bestlifeonline.com/rude-to-wear-a-hat-indoors/

It’s an ancient custom dating back to the days of knights, and generally courteous when visiting some one new or entering any professional establishment.