Racism.

Thats literally it.

HoA’s became a major thing after WW2, in large part, to keep black people from moving into white neighborhoods… with rare exceptions for “servants”.