[4] Fraser, born in Blackford, Perth and Kinross, began his football career as a youngster with Gairdoch United before joining Dunfermline Athletic in 1958 at the age of 17.
[3] Under Jock Stein's management at Dunfermline, he was a Scottish Cup-winner in 1961, defeating Celtic 2–0 in the replay after the original tie had finished goalless.
[3][10] In October 1962, Fraser joined English First Division club Aston Villa for a fee of £24,000,[11] which paid for Dunfermline's new main stand.
[7] He played 40 games for Aston Villa in all competitions, and was on the losing side in the 1963 Football League Cup final against Birmingham City.
Following a contractual dispute with the club he quit, joining his wife's hairdressing business in London, before returning to football in February 1965 with Birmingham City, who paid Villa a £9,000 fee for his services.