Yeah, Bethesda loves to ruin their game worlds with weirdly repetitive additions. Morrowind constantly spawns assassins on you, Oblivion does the Oblivion gates, Skyrim has the dragons. In the latter two, I think, it’s best to just not start the main quest, which prevents the Oblivion gates and dragons from appearing, at least if you replay the game.