This is an extremely old philosophical idea (It's commonly referred to in question form as "Is my blue your red?), and unfortunately, there are very few ways of testing it. The best we can do is use a proxy test. In the case of color, we ask "Which is color closer to blue: red or yellow?"
That's not a very good test, though, because some would say that yellow, being a lighter color than the other two, is not closer to red than blue is. Others would say that red and yellow are closer colors, because they are both warm colors, while blue is a cool color.
If we delve into evolution and genetics, however, we can conclude that people do taste things somewhat differently, but the differences are often subtle. These are in practice more differences in the reaction to our perception in the taste, smell, feel, etc. of something. For example, many people dislike spinach, but I enjoy it, as long as it's cooked in something.
One similar condition is that in all humans, the color green is the most restful to our eyes, and yellow blends in with white easier than blue or red do. These facts may show that perception of color is close in all people. The same applies for the fact that in fashion and art, the same colors complement each other for all people. Blue always pops out in front of red, and green and red are always opposites.
In my opinion, it is highly unlikely that one person's red is another's blue, but the perception of colors is likely to be somewhat different between people.