Depends, at some points you need enough to progress. I might have been a little too controlling of my cult’s population for my playthrough, because there were several times I struggled a bit to meet the requirement.

Haven’t bought the latest DLC yet. I might return to it.