The second son of Ralph and Catharine Gough of Gosbrook House (later Gorsebrook House), Bushbury, Staffordshire, he was christened on 7 February 1825 at the church of Saint Peter, Wolverhampton, Staffordshire.
He was educated at St John's College, Cambridge, where he obtained a BA in 1847 and his MA in 1853.
He married Mary Vigars LeMare, at Christ Church, Salford, Lancashire, on 4 April 1854; he was widowed seven years later when his wife died in London in early 1861.
He then married Ann Marie, the widow of the Reverend John Jones who had been a missionary in Ningbo (Ningpo).
They were married at Trinity Church, Bow, London on 15 November 1866.