Well, the Falmer were there first really. They were subsequently enslaved by the Dwemer. Then the Atmorans (Nord) came along and performed a genocide and have been butthurt about it ever since.