I tend to play without expansions and mods the first time. The second time, with expansions. Later, with any mod I specifically want.
The exception was Daggerfall because Arena was pretty bad with a few bugs. However with Daggerfall Unity I didn’t install additional mods. I did look for additional bug fix mods, didn’t find any. That makes sense since if an external modder makes fixes, the Daggerfall Unity maintainers would make those fixes too eventually. I need to look again though.