I had similar issues until I gave myself this rule: only do side stories and main story quests. If they require playing a mini game, sure, but only play the minigames as a novelty, don’t try to complete them!