He was the second-leading rebounder for Kansas' 1953 national runner-up team that lost the NCAA Men's Division I Basketball Championship game by a single point to Indiana University.

)[2] Known as "Prince" Hal, in 1956, he won the Jeff Russel Memorial Trophy then the Schenley Award as the Canadian Rugby Union's Outstanding Player as a tight end.

Patterson was a member of the Alouettes until being part of a controversial trade in 1960 that sent him to the last-place Hamilton Tiger-Cats with fellow Montreal star quarterback Sam Etcheverry.

Patterson's impact was immediate, as he helped to lead the Tiger-Cats to the 1961 Grey Cup, where the Ti-Cats lost in overtime to the Winnipeg Blue Bombers.

In November 2006, Patterson was voted one of the CFL's top 50 players (#13) in a poll conducted by Canadian sports network TSN.