In 1701, Lord Lovelace married Charlotte, the daughter of Sir John Clayton,[1] but her poor dowry little improved his financial situation.

The couple had six children: John (d. 1709), Charlotte (d. 1705), Charles (d. 1707), Wentworth (d. 1709), Neville (d. 1736) and Martha (d. 1788), who married Lord Henry Beauclerk in 1739.

However, in 1705 he was described by a Scottish writer named John Macky as follows: "Lieutenant Colonel of the Horse Guards; a very pretty gentleman of good sense and well at court; a short, fat brown man, not forty years old.

After a harrowing nine-week trans-Atlantic voyage Governor Lovelace, his wife and three sons John, Wentworth, and Neville arrived in New York on 18 December 1708.

His widow and two-year-old son Neville returned to London, where Lady Charlotte gave birth to Martha who was baptised in Westminster on 14 January 1710.