They do actually yeah, the nest allows moisture into the wood it’s attached to.

I was in a rental that was a shitbox, so I doubt my LL would have even noticed tbh. But if you’re concerned about damage to the wood (wherever they are attached), then yeah you’d need to remove them.