He was the fifth and youngest son born to John Drummond, 2nd Earl of Perth, and Lady Jean Ker.
As both of his maternal uncles predeceased his grandfather, the 1st Earl of Roxburghe, without a male heir to inherit his titles and estates, nominated his grandson William for the honours in 1648.
William changed his surname to Ker and, by special arrangement ratified by Parliament in 1661, inherited the titles and estates of his grandfather.
Harry Ker (from his grandfather's second marriage to Jean Drummond, who was also his father's younger sister) and Lady Margaret Hay (the only daughter of William Hay, 10th Earl of Erroll and Lady Anne Lyon, daughter of Patrick Lyon, 1st Earl of Kinghorne).
Together, they were the parents of:[5] Roxburghe died on 2 July 1675 and his titles were inherited by his eldest son, Robert.