[1] Harley was the eldest son of Edward Harley, 3rd Earl of Oxford and Earl Mortimer and Martha Morgan, daughter of Welsh politician William Morgan.

[2] He was educated at Westminster School between 1735 and 1744; and then Christ Church, Oxford, graduating with a Doctor of Civil Law in 1748.

Prior to the election, Lord Foley wrote to Harley's father, "As he is under age I am in great fear lest some trick should be played on him on the day of election ... which if it should happen I think would be of the most evil consequence to your family as well as to the interest of the county.

[4] In 1751, he married heiress Susannah Archer, who brought a dowry of £50,000 (equivalent to £9,923,407 in 2023).

[6] As he had no children, his titles and estates passed to his nephew Edward Harley.