Do you have a particular edge case in mind? One that’s commonly brought up is en.wikipedia.org/wiki/Ovotesticular_syndrome, but that doesn’t fall outside the sex binary. Having a bit of nonfunctional tissue doesn’t affect one’s sex.

Colors aren’t a great analogy either, because in anisogamous species, gametes are strictly binary. There’s sperm and ova, with 0 overlap and 0 other options. “Purple gametes” just don’t exist.

This also isn’t my opinion, this is the accepted definition in the field of biology.