Lower difficulty, in all present difficulty options the raid strength is determined largely based on your colony wealth. If you have massive towers of gold then expect big raids.

Even flooring adds up to wealth, massive rooms with carpets can become a non insignificant value. This isn’t to say don’t build them, but you should make sure a reasonable amount of your total wealth is invested into warfare. It can be quite easy to hoard things that are useless to you but have a fair bit of value to them.