Economic meltdown, most of the population has no one that would take them so they will end up either in camps in failed/failing countries or inside the US itself.

Also getting the white population to agree to do it would be a big lift. The majority would passively oppose and a minority would go full on underground railroad/insurgency.

Only a minority would be totally on board.