[2] Olson was born in Honolulu, Territory of Hawaii to a Portuguese mother and a Swedish father, hence his other nickname "The Hawaiian Swede".

Olson received training from boxers who were stationed in Hawaii during World War II, and it was during this period that he picked up his two trademark tattoos.

Seven months after this Olson had his first fight against Sugar Ray Robinson, for the lowly regarded Pennsylvania State World Middleweight Title.

As the fight progressed Olson took the initiative, he scored knockdowns in the 10th and 11th rounds on the way to a unanimous decision.

[3][5] Olson won all seven of his fights in 1954 including defenses of his title against such big names as Kid Gavilan, Rocky Castellani, and Pierre Langlois.

[3] In 1955 Olson, who was finding it increasingly difficult to make weight, stepped up to light heavyweight.

[3][2] Following two walkover wins, Olson put his middleweight title on the line against Robinson, who was once again number 1 contender following his brief retirement, on December 9, 1955.

It was Olson's fourth and final defense of his title, but ended as a stunning comeback for Robinson.

The rematch, fought five months later at Wrigley Field, on May 18, 1956, ended similarly with Olson going down in the fourth.

Whilst initially coming back as a journeyman, despite being only 30, Olson managed to reestablish himself as a contender.

[3] Olson retired with a record of 97 wins (47 by KO), 16 losses, and 2 draws from his 115 professional fights.

In 1987 he was a Union Elevator Operator in Lancaster, California, working on new construction at the Antelope Valley Medical Center.