[2][3] The house is a large three-story wood-frame structure, set on a lot roughly 1 acre (0.40 ha) in size, built in 1791 for Colonel Joseph Nightingale.

In 1853 John Carter Brown hired Thomas Tefft to design a carriage house for the property, and in 1858 a two-story addition was added to the south side.

[3] In 1862 Brown hired Richard Upjohn to design a library wing, which was built in 1862–1864 and is northeast of the main block and forms a courtyard with the carriage house.

He was a leading industrial developer of the early 19th century, and was such a benefactor of the local college that it was renamed Brown University in his honor.

The John Nicholas Brown Center for Public Humanities and Cultural Heritage (JNBC) was housed in the building for many years.

The center, founded as the John Nicholas Brown Center for the Study of American Civilization in 1979 and part of Brown University since 1995, collaborates on public and scholarly humanities projects with local, national, and international partners working in the arts, history, education and community engagement.