To be fair, I don’t think it’s “ridiculous” to sort e.g. jeans into the broad categories of “typically wider or slipper hips/thighs compared to length” or t-shirts into “typically broader back vs. typically larger chest”.

The mens/women’s categories are probably the coarsest categories that makes sense, since the average man’s and women’s body are so different in so many ways.