It’s not only Apple Photos. Google Photos does the same shit but at least for the latter (don’t know if Apple offers this feature) there is an option to exclude people from these “highlights”.