If the colours are arbirary, have they ever been different? Well, it turns out, yes, in fact they have:
The Sunday Sentinal, an American newspaper, in 1914 advised mothers: “If you like the color note on the little one’s garments, use pink for the boy and blue for the girl, if you are a follower of convention” (March 29, 1914).
Similarly, Ladies Home Journal informed: “There has been a great diversity of opinion on the subject, but the generally accepted rule is pink for the boy and blue for the girl. The reason is that pink being a more decided and stronger color is more suitable for the boy, while blue, which is more delicate and dainty, is prettier for the girl” (June, 1918).Alas, no tangerine for boys, but Pink is definitely a manly colour. Who can argue with logic such as theirs? And note the dates: 1918 was not a very long time ago, in the grand scheme of things.
As you can probably guess, I'm quoting from a paper on the subject, which started out so great as it goes into the history, and the flip-flop somewhere in the early-mid 1900s. Oh, but it started out so great, ends in a great big wallow in Evolutionary Psychology. Evo-Psych can be a good tool for exploring behaviour, when employed properly, and in a comparative context. It tries to explain human behaviour in an evolutionary (often adaptive) context, bringing examples from our past.
But often, when it's implemented, it's vacuous nonsense. There's nothing to it. It's a lovely story that sounds great, but is ultimately untestable. They're sometimes called "Just So" stories, echoing a bit of the lack of rigour to them (and referencing a child's book). Why do humans sneeze when they go out into the light? Because it helped to clear our noses in our cave days. It's just so. This pseudoscience is rightly derrided. And Frassanito and Pettorini dive into it deep:
A recent argument proposes a biological basis, connected to evolved sex differences in specialized visual pathways that allows females to better discriminate red wavelengths. The hunter–gatherer theory proposes that female brains should be specialized for gathering-related tasks and is supported by studies of visual abilities . Tricromacy and the second red–green system (L–M opponent channel) are “modern” adaptations in primate evolution thought to have evolved to facilitate the identification of ripe, yellow fruit or edible red leaves embedded in green foliage . It is therefore plausible that, in specializing for gathering, the female brain honed the trichromatic adaptations (and developed more the P-cell pathway of vision), and these underpin the female preference for objects reddish. Research on foraging in contemporary nonhuman primates  supports this hypothesis. Whereas discrimination of red wavelengths appears to facilitate identification of plant food, a preference for red or pink appears to have an advantage for successful female reproduction. This preference for reddish-pink is thought to exist because infant faces compared to adult ones are reddish-pink, and red or pink may signal approach behaviors that enhance infant survival Where do I begin with that? First of all, it assumes a sexual segregation in hunting/gathering. Did women gather more than men? I'll dispute this in a moment, but let's assume for a moment that they did. So selection would favour females who had a red-preference. But. There is no down side to males inheriting this preference too. There's no reason for it to become sex-linked. This is good, because it may be highly difficult for such a behaviour to become sex linked! It would need to become tied to some sexually differentiated system, and this adds an additional layer of complexity. No one would deny that sex based differences in the male and female brains exist, but most of the big ones are older differences (such as mate preference).
And that assumes there was sexual segregation in labour! I had the good fortune of studying under Dennis Bramble for a short while, and he'd reported that in his studies of human adaptation to cursoriality (running), he found no sex differences for the middle percentiles. The elite athletes have a sex difference, but athletes are, by definition, weird. They're not normal. He reported to his class there was no sex difference human running energetics until some point deep into pregnancy (I don't recall the exact point, but it surprised me). As cursorial mammals, there is a minimum of sex-based differences. So much for evolved sexual segregation of labour!
It's worth nothing that  doesn't actually support their hypothesis AT ALL. There is no mention of sex based differences. At all. Anywhere. How does that support them? They might argue "Well, human (American) males don't show the preference! Ergo, there has been differentiation." Yes, but how many human males in America actively forage? The trait is superfluous, so if it is under cultural control, it could vanish without anyone being harmed. In fact, the sexual dimorphism in "Westerners" suggests that it is not an evolved characteristic, in light of our primate relatives!
So, where the heck is their evidence?
Frassanito, P., & Pettorini, B. (2008). Pink and blue: the color of gender Child's Nervous System, 24 (8), 881-882 DOI: 10.1007/s00381-007-0559-3