He was born on 5 September 1839 to James Henry Robert Innes-Ker, 6th Duke of Roxburghe and Susanna Dalbiac, Duchess of Roxburghe, only child of Sir Charles Dalbiac.

[1] His elder sister was Lady Susan Harriet Innes-Ker, who married Sir James Suttie, 6th Baronet.

His younger siblings were Lady Charlotte Isabella Innes-Ker, wife of George Russell, and Lord Charles John Innes-Ker, who married Blanche Mary Williams (a daughter of Col. Thomas Peers Williams).

Together, James and Anne, who served as Mistress of the Robes and Lady of the Bedchamber to Queen Victoria, were the parents of seven children, three boys and four girls, including:[2] Lord Roxburghe died on 23 October 1892 and was succeeded by his eldest son.

[2] His widow, the Dowager Duchess of Roxburghe died in London in 1923 after a lengthy illness, at the home of her daughter Lady Evelyn and her husband, Colonel William Collins.