[2][3] Harsson's daughter, Laura, married Charles Wesley Goyer, who bought the house in 1852.

Goyer had it expanded by the architecture firm of Edward Culliatt Jones and Matthias H. Baldwin in 1871, after seeing their work in designing the neighboring Woodruff-Fontaine House.

James Lee, a riverboat captain who had been educated at Princeton University, bought the house in 1890.

[5] It was used by Canadian indie rock group Tokyo Police Club in a music video for their 2008 song "In a Cave.

[6] The following year, a $2 million construction and renovation project began, converting the house into a luxury bed and breakfast.