Depending on what you are using to generate the images. I have only used AUTOMATIC1111 and you can set a specific seed for the person being generated separated from the regular rng seed.

Alternative you could use img2img or inpainting to change specific parts of the generated image.