His mother was quite wealthy as her father eventually became the sole heir to estates held by James Hamilton, 1st Earl of Clanbrassil, whose male line had died out with his son.
In 1799, James' father died, leaving him to inherit the baronetcy as his elder brother, Robert Blackwood, had been killed in 1785 in a fall from his horse.
[2] In 1800, his mother was created Baroness Dufferin and Claneboye of Ballyleidy and Killyeagh in the Peerage of Ireland, with remainder to her heirs male.
[1] He was later elected to the British House of Commons, serving as Member of Parliament for the Cornish constituency of Helston from 1807 to 1812, and for Aldeburgh in Suffolk from 1812 to 1818.
He had been under treatment by Sir Henry Halford in London and then spent a week at the home of his brother-in-law, Thomas Skeffington, 2nd Viscount Ferrard, in County Louth.