Pansexual still refers to the gender. Same with bisexual and straight people. One might be more specific like “only cis straight” (I know it sounds horrible, but couldn’t think of a better naming), but they all generally apply to the gender when used as broader terms.