Historically (and still today) ice/border patrol has been in cahoots with the farmers and businessmen to suppress/steal wages from immigrants.
Pulling lots of people from the local economy is going to depress said economy. So yes, ruthlessly yanking people will screw everyone. Especially the working class.