Both are relevant, absolutely.

unless you were lucky enough to only go through one puberty, your body has went through some irrrversible changes

For transsexual men, the only irreversible change from first puberty is impact on bone structure/stature. Being smaller is not an exclusively female trait though, or 5’2” cis men would not exist.

Not to disagree that birth sex isn’t any more medically relevant than gender though, and all of that should be part of your medical record and doesn’t need to be elsewhere for that purpose.

Yep, this.