Built in 1809–10, they form the largest surviving brick building from the Federal period in the city.

[2] Brown Square is a narrow rectangular park at the northern end of a city block on the west side of Newburyport's downtown, bounded by Titcomb, Pleasant, and Green Streets on the west, north, and south, and by Brown Square on the south.

The Brown Square House occupies the western end of the block facing the park to the north.

[2] The two rowhouses were built in 1809 and 1810 by Moses Brown, a Newburyport landowner, shipbuilder, and shipping merchant.

In 1922 it was turned into a hotel, named in honor of abolitionist William Lloyd Garrison, a statue of whom graces the square.