On March 9, 1820, they were married in the White House, in a small ceremony officiated by Reverend William Hawley, with only 42 guests in attendance.

Hay, who acted as an unofficial First Lady during the Monroe presidency, created a social scandal by trimming the guest list and snubbing large portions of Washington society.

They planned a series of lavish balls for the couple at Decatur House to appease complaints.

[2][5] In 1822, the couple moved to New York City, where Samuel Gouverneur held a series of political posts.

[2] After his wife's death in 1830, James Monroe lived with his daughter and her husband in Prince Street, New York City[10][11] until he died on 4 July 1831.