He was the son of Lady Augusta Vane and Mark Milbank, MP for Camelford and High Sheriff of Yorkshire.

[6] In 1842, Milbank inherited estates at Hart and Hartlepool in county Durham from his maternal grandfather, the 1st Duke of Cleveland.

[2] As a Liberal,[8] Milbank was elected to the House of Commons for the North Riding of Yorkshire in 1865, a seat he held until 1885, and then represented Richmond until 1886.

[9] In 1844 Milbank married Alexina "Aline" Harriet Don (c. 1826–1919),[10] daughter of Sir Alexander Don, 6th Baronet and the author Grace Jane Stein (a daughter of Edinburgh banker and distiller John Stein).

[16] Reportedly, Sir Frederick had a child, named Madeleine Augusta Crabb, out of wedlock with the stage actress (and artist's model to Dante Gabriel Rossetti) known as Ruth Herbert (née Louisa Ruther Herbert Maynard).