The Mastership of the Buckhounds being a political office, the holder changed with every government and because the Earl's patrons fell in March 1807 he lost his position after only one year.

The Earl was accorded the honour of travelling to Westminster Abbey inside the Gold State Coach with the nineteen-year-old, and as yet unmarried Victoria, who recorded in her diary: "At 10 I got into the State Coach with the Duchess of Sutherland and Lord Albemarle...It was a fine day, and the crowds of people exceeded what I have ever seen; their good humour and excessive loyalty was beyond everything, and I really cannot say how proud I feel to be the Queen of such a nation".

In addition to managing the bloodstock of two successive heads of state, when the horse was still a main mode of transport, Lord Albemarle was also a leading racehorse owner of his day.

One of the witnesses of this triumph, Tsar Nicholas I of Russia, let William Charles know how excited he had been by the race, and the Earl promptly named his horse "The Emperor" in honour of the distinguished Russian visitor.

They had at least twelve children: After his first wife's death in November 1817, aged 41, Lord Albemarle married, secondly, Charlotte Susannah, daughter of Sir Henry Hunloke, 4th Baronet, on 11 February 1822.