Plus the only places that are left to build giant factories are distant from population centers.

You’re forgetting about eminent domain. Clearly, giant factory on American soil is a public good, even if it’s privately owned. Hence, the government will seize the homes and give the land to the corp. The prices of the houses will fall first, because who wants to buy a house that’s going to get bulldozed, reducing the compensation the government has to pay to the homeowners, potentially causing them to be upside down on their mortgages. This will leave them with nothing, or potentially even debt, when their homes get sized.