All my ssh keys are password protected, so I personally dont have a difference there. I would recommend doing that as a protection against supply chain attacks that steal keys.