Scientific research indicates we see colors pretty damn similarly, with edge cases for colorblindness and also people who are more color sensitive.

One way this can be studied is by studying the metamerism of different colors by different observers. Metamerism is the study of how colors change given different light sources.

There are other objective qualities that give hints that we have similar ways of experiencing colors. You mention that colors are nothing more than our brain assigning “color” to frequency of light – but light is itself just a frequency of electromagnetic radiation, namely the frequencies that make up the bulk of the radiation emitted by the sun.

So to a normal observer without colorblindness, there are more variants of colors of green than any other color. Green is of course situated in the very center of the roygbiv spectrum, it is the “most visible” color. The colors with the least amount of variations are red and violet, which are situated at the edges. Frequencies above violet or below red become invisible making up infrared and ultraviolet radiation.

Where we get tricked up, and I used to have identical suspicions as you did, is that we consider color to be purely subjective, because we aren’t taught to unify subjectivity and objectivity into a united whole. Color isn’t completely imagined, there are certain surfaces that absorb and reflect certain frequencies of EM radiation just as the structures in our brain that process this ocular input are more or less similar. Things that are subjective aren’t usually associated with being “real” the same way that objectively “real” things that exist out in the phenomenal world are. However, color is socially real, we can almost all identify colors that are the same and colors which are different. Since the set of colors which are “red” are fewer than the set of colors which are “green” then there is no way that what I experience as red is the same as what you experience as green. Artists use colors to convey emotion and are able to achieve this with many many different observers. Warm colors are warm, and cool colors are cool. There may be different levels of sensitivity but in my experience this can be somewhat trained into an observer though no doubt there are outliers who have a unique sensitivity to color differences.

So there are objective factors which align with subjective factors let’s say 90% of the time, which strongly supports the idea that we experience color more or less the same way. The trouble is not that subjectivity and objectivity are irreconcilable, in fact it is when we fail to reconcile them that our troubles begin. In my opinion, this is a huge problem that creates all kinds of issues when we try to relate to each other; it may be the most prominent philosophical problem of our age. Luckily it is fairly easily remedied with a slight change in the way we think about subject and object. Its useful to separate them sometimes but we need to be able to reunify them, which just takes practice in my experience.