I think it’s valid either way. “I’m not attracted to men, thus I’m straight” is as valid as “tried it, didn’t like it, thus I’m straight”

And works the other way around too, I imagine many people who define themselves gay have had romantic or sexual experiences with the opposite sex, but didn’t like it.