The house was built on the highest point in Essex County on land granted in Nicholas Holt (1).

[2] The first Holt in Andover was Nicholas Holt (1), an early officer of the town, who was a tanner who also ran the ferry across the Shawsheen River.

Despite some evidence that Nicholas himself was barely literate, his descendants became known for their academic accomplishments with a long line of ministers and teachers.

[3] Early settlers of Andover, some the Holts, who settled on the "Stoney Plaine", west of Holt Hill, lived near Scotsman Robert Russell (1) 1630–1710., in the part of Andover long known as the 'Scotland District.

[4] Russell, the first person to be interred in the newly designated South Parish burying ground in 1710, had ten children, three of whom married Holts from nearby Holt Farm.