He was educated in Oundle School and Trinity Hall, Cambridge, with an interruption of four years, when Milner served in the Royal Air Force as a Flight Lieutenant.
He graduated with a Master of Arts in 1948, working then as solicitor in his family firm.
Milner was Assistant Labour Whip from 1971 to 1974, and retired when Harold Wilson became prime minister again.
He was further a Member of the Worshipful Company of Clothworkers and Honorary Treasurer of the Society of Yorkshiremen in London between 1967 and 1970.
They had three children:[1] After his wife's death in 2000, Milner married secondly Helen Cutting Wilmerding in 2002.