ncumbent PM made joke about Rishi Sunak’s department because of its focus on lifting restrictions, according to diary entries