Goldstone was born in Bishopwearmouth, County Durham (now Sunderland) on 7 December 1870.

The third son of a stained-glass artist, he attended Borough Road Traininge College, Isleworth after completing education at Diamond Hall in Millfield.

A member of the National Union of Teachers (NUT), he became president of the subgroup National Federation of Class Teachers in 1902, a member of the executive committee of the NUT in 1902 and chair of its law committee in 1904.

Shortly before the December 1910 election, Goldstone was chosen as the Labour candidate for Sunderland, as a late replacement for R. J. Wilson (a member of the Co-operative Society).

Goldstone was knighted by George V at Buckingham Palace on 29 June 1931, the same year he retired as general secretary of NUT.