He was the youngest son of Lady Elizabeth Montagu (1743–1827) and Henry Scott, 3rd Duke of Buccleuch.
His elder siblings were Lady Elizabeth Scott (wife of Alexander Home, 10th Earl of Home), Lady Mary Scott (wife of James Stopford, 3rd Earl of Courtown), Charles Montagu-Scott, 4th Duke of Buccleuch,[1] Lady Caroline Scott (wife of Charles Douglas, 6th Marquess of Queensberry).
His younger sister was Lady Harriet Scott (wife of William Kerr, 6th Marquess of Lothian).
[2] On his maternal grandfather's death in 1790, when the Dukedom of Montagu became extinct, the Barony passed under a special remainder to Lord Henry Scott.
[3] Lord Henry also inherited the Lordship of Bowland from his mother, the Duchess, having the title entailed upon him by his nephew, the 5th Duke, in 1827.