Although Tories claim curbing net inflow of migrants is critical issue for voters, poll shows attitudes have evolved significantly