He came of an old Anglo-Irish family settled in Ireland since the days of Sir Faithful Fortescue (1581–1666), whose uncle, The 1st Baron Chichester, was Lord Deputy.

The history of the family was written by his elder brother, Thomas Fortescue, who in 1852 was created Baron Clermont.

The future Lord Carlingford was educated at Christ Church, Oxford, where he took a first in Classics (1844) and won the chancellor's English essay (1846).

He was admitted to the Imperial Privy Council in 1864 and the following year he was made Chief Secretary for Ireland under Lord Russell, a post which he again occupied under William Ewart Gladstone from 1868 to 1871 (this time with a seat in the Cabinet).

Lord Carlingford married Frances Elizabeth Anne, Countess Waldegrave, daughter of John Braham, in 1863.