A Fabian and a member of the Labour Party, he served as Governor of Jamaica and as Secretary of State for India in the first government of Ramsay MacDonald.

[1] His brothers included Henry (1850–1935), who had a military career ending as a colonel,[2] Herbert, a successful portrait painter, and Gerard (1869–1939), a clergyman (the father of Laurence).

After graduation Olivier resisted family pressure to train as a barrister and instead sat the competitive examination for the Civil Service.

He was attracted to the Positivist vision of a moral reform of capitalism, rather than mere amelioration, and for a while entertained this notion as an alternative to socialism that might be more palatable to Victorian England.

In 1891 the Oliviers made a permanent home in Limpsfield; several other Fabians and radicals moved to the area, and they soon became the dominant force on the parish council.

In 1892, Olivier and Shaw attacked Robert Blatchford, Fabian leader in Manchester, for calling for members to boycott both the Conservative and Liberal parties at the ballot, regardless of the policies of individual candidates.

Other Fabians, including Webb and Shaw, believed military action could be used to promote democracy and civilisation, whilst some also felt that the best policy was to reform the British Empire rather than, as Olivier advocated, retreating from it.

Having caused a stir in Downing Street by voicing his opposition to the war and his criticisms of Chamberlain, Olivier was posted as Colonial Secretary in Jamaica, departing in early 1900.

This posting ended in 1903 and Olivier returned to England, but he went back to Jamaica a short while later to work in relief and rehabilitation following a devastating hurricane, and served as acting governor for a third time.

Moving outside of the Colonial Office, he served as Permanent Secretary to the Board of Agriculture and Fisheries for four years, then as assistant comptroller and auditor of the Exchequer from 1917 until 1920, when he retired from the civil service to devote himself to philosophical and political study.

Under Olivier there was no departure from the Conservative policy on India, although Mahatma Gandhi was released from prison after serving only two years out of a six-year sentence.

Privately, he believed that the problems of India could not be solved at that time or by a minority Labour government, and resolved to merely defend the status quo.

In July 1924, he sided unsuccessfully with Wedgwood, Philip Snowden and J. H. Thomas in the Cabinet, opposing the promise of a loan to the Soviet Union, and was critical of MacDonald's decision to call an unnecessary election later that year.

Four daughters were born to the couple: They were prominent in the Cambridge and Bloomsbury social circles around Rupert Brooke and in what Virginia Woolf dubbed the Neo-Pagans.