Yes.

Not only due to privacy concerns (my main concern), as the device will constantly ping home even when turned off. Other concern is it will download “updates”, that eventually may render your tv browsing experience laggish.

Some tv sets have not only mics incorporated, but cameras, so it may depend on your level of concern.