Turns out I was wrong about modifications being needed since linux-xtensa has now been merged into mainline linux: Kernel Github Repo