The Federal style house was originally built in 1790 on the banks of the Middlesex Canal, for Woburn lawyer Joseph Bartlett.

Shortly before completion it was purchased by Col. Loammi Baldwin,[2] noted engineer, who hoped to convince expatriate scientist and inventor Benjamin Thompson, Count Rumford, to return to his home town.

Although this idea never came to fruition, author Frances Parkinson Keyes, who later spent childhood summers in the home, refers to it repeatedly in her memoirs as the Count Rumford House.

In 1815 Hall Jackson Kelley conducted a private school for boys in the house, and there he first read the newly published Journals of Lewis and Clark.

Kelley conceived a passion for the Pacific Northwest and became the prime advocate of the United States settlement of Oregon.