On succeeding his father at Hardwicke Court in 1841, he took an active part in the administration of other local public institutions, was one of the founders of the Social Science Congresses, started what is known as the Berkshire system for the suppression of vagrancy, was president of the chamber of commerce, and captain of the Gloucestershire squadron of the yeomanry cavalry.
Baker's attention had been drawn to the question by seeing boys in prison at Gloucester, and by a visit to the Philanthropic Society's school in London.
In 1861, the whole question of the treatment of youthful offenders was considered at a conference at Birmingham, promoted by the town clerk, William Morgan, and Joseph Hubback of Liverpool.
With the help of George Henry Bengough (1829-1865), Baker opened a school at Hardwicke in March 1852, the first inmates being three young London thieves.
[1] Many particulars of Baker's work are given by Professor von Holtzendorft, who made his acquaintance in 1861, and published a book which was translated by Rosa Gibhard under the title, An English Country Squire, as sketched at Hardwicke Castle.
A collection of Baker's papers, contributed to newspapers or read at meetings of the Social Science Association, was after his death edited by Herbert Philips and Edmund Verney in 1889, under the title, War with Crime.