There are three distinct concepts I think you’re confusing:

• The idea of biological races. Yeah, a given culture’s definition of “race” is historically contingent and biologically incoherent. I think you get that and are assuming that’s all there is to it.

• Race as a correlative of ethnicity. There are some ethnicities whose members tend to have darker skin colors, and people tend to conflate skin color and ethnicity. Ethnicity (as a set of cultural institutions) is meaningful to some people, and some of them interpret a disregard for “race” as a disregard for their ethnicity, or as an attempt to suppress ethnic identity.

• Race as a social construct. When the above ideas permeate a society, people with different skin colors experience systemically different treatment—even in the absence of actual biological or ethnic distinctions. So people with similar skin colors can be grouped on the basis of those shared experiences, and the different behaviors resulting from those experiences feed back into the society’s conceptions of biological race and ethnicity. And it doesn’t suffice to counteract such social constructs by ignoring them—social behavior is taken for granted unless people make a conscious effort to reevaluate it.