Construction on the castle, including addition of a crenellated tower, continued through the Victorian era, when it was considered an engineering marvel.

Founder of Brownsville (Thomas Brown) sold the land to a man named William Lynn in 1786.

Likely more than 1,200 years before construction of the fort, trading post or castle, prehistoric indigenous peoples had also found this site a strategic one.

Jacob Bowman and his wife started building the first part of the structure sometime around 1789 (before gaining full ownership of the land in 1795), with a trading post on the ground floor and one room above.

When Nelson died in 1892, he left the house to their son Charles Bowman, who lived there with his wife Leila until his death.

By her will, after the widow Leila Bowman died; years later The National Historical Society purchased the house and began to open it up to the public as a museum.