He spent some time in Labuan at first as James was establishing himself as its first governor, but was subsequently based in Kuching.

He has been largely ignored in the standard historical accounts of Sarawak, but substantial records survive [4] which show how active he was, and his engagement with pirates in the Battle off Mukah in 1862 has attracted interest.

Brooke married Anne Grant, a granddaughter of Lord Elgin, at Kilgrastron in Scotland in 1856.

Brooke's second marriage was to Juliana Caroline Welstead: they had met in England and married in Singapore in 1861; Julia died a year later giving birth to a daughter, Matilda Agnes (1862–1943).

Brooke died in Hounslow in 1868 after a long illness, and was buried in his father's churchyard at his childhood home of Whitelackington.