Following his father's death, James travelled to Jamaica where, like his brother John Wedderburn, he made a fortune in the sugar trade.
In 1773, he returned to Scotland where he married the heiress Isabella Blackburn in 1774, and purchased a country seat at Inveresk Lodge; raising a family of six children.
In his later years, Wedderburn lived the life of a country gentleman (adding the extra name Colvile for reasons of inheritance) until his death in 1807.
[2][3] James Wedderburn returned to Scotland in 1773; he married Isabella Blackburn in 1774 and bought Inveresk Lodge, Midlothian, where he lived in comfort until his death in 1807.
In his will, dated 1802, the bulk of his estate was left in trust to his eldest legitimate son Andrew Wedderburn Colvile.