He left the island in 1849 to become the private secretary of George V of Hanover, leaving that office in 1855 when he succeeded to his father's title.

He was commissioned as a Lieutenant in the disembodied North Hampshire Militia on 31 October 1849,[5] transferring to the Gloucestershire Yeomanry from 1856, initially as a Cornet and then as a Captain 1864–75.

An inscription over the porch dated 1858 reads: "This house with 238 acres of land was purchased by 1623 of the friends, admirers and comrades in arms of the late Field Marshal Lord Raglan GCB and presented by them to his son and his heirs for ever in a lasting memorial of affectionate regard and respect".

On 25 September 1856 Somerset married Lady Georgina Lygon, the third daughter of Henry Lygon, 4th Earl Beauchamp and Lady Susan Caroline Eliot (a daughter of the 2nd Earl of St Germans).

Together, they had one daughter:[1] Lord Raglan died in Chesterfield Street, London, on 3 May 1884, and was buried in Llandenny churchyard, Monmouthshire, on 8 May 1884.