Spencer made his League debut in 1950, but his career really took off with the arrival of Norm Smith as coach of Melbourne for the 1952 season.

His contemporaries included prominent players of that era – Ron Barassi, Brian Dixon, (Frank) 'Bluey' Truscott, Johnny Beckwith.

He was recognised as an outstanding player, winning back-to-back best and fairest awards for Melbourne in the premiership years of 1955 and 1956, as well as being the club leading goalkicker in 1955.

It was thus highly surprising that, at the peak of his VFL career, Spencer moved to Tasmania and became captain-coach of Clarence in the Tasmanian Football League in 1957.

At age 74 Spencer still worked as the managing director of the Greens Group, a Hobart removals and storage company.