[2] He ran the Pembroke family estates, centred at Wilton House, Wiltshire, for most of his adult life.

The hard work entailed caused a breakdown in his health, so that in July 1861, having been created a baron in the peerage of the United Kingdom, he had to resign government office.

In the early 1840s, Herbert is thought to have had an affair with the noted society beauty and author Caroline Norton, who was unable to get a divorce from an abusive husband, so that the relationship ended in 1846.

After her husband's death, Lady Herbert became an "ardent ultramontane"[citation needed] Roman Catholic, along with their eldest daughter, Mary.

In New Zealand, the highest peak on Banks Peninsula was named Mount Herbert by the chief surveyor of the Canterbury Association, Joseph Thomas, in 1849.