In 1888, he married the Dowager Duchess of Montrose, a distant relation 46 years his senior (they were both descended from Marcus Beresford, 1st Earl of Tyrone).
In 1874 after the death of her first husband the James Graham, 4th Duke of Montrose, she had married a wealthy racehorse owner, William Stuart Stirling-Crawfurd (1819-1883); she became an established breeder and trainer of racehorses, though forced by the conventions of the time to use a male pseudonym, "Mr Manton", for the business.
For some time after her marriage to Milner the horses ran under his name, though the whole business had been sold in the months before the Duchess's death in 1894.
[3] According to his own short obituary in The Times, Milner acted as "controller" to successive generations of the Stanley family, the Earls of Derby.
[6] He volunteered for service as a major in the First World War, when he was aide-de-camp to the commanding officer of the 55th (West Lancashire) Division and was awarded the Belgian Croix de Guerre.