Built in 1883 to a design by John Calvin Stevens, it is one of the city's few examples of Queen Anne Victorian commercial architecture.

It is a large 4-1/2 story masonry structure, built primarily out of brick, occupying about 2/3 of the block between Casco and Oak Streets.

Its main facade is 120 feet (37 m) long, with four store fronts which are slightly stepped due to the sloping lot.

Near the center of the facade is the main entrance, recessed behind a stone arch; the bays above this entrance include a round-arch window on the third floor, and it is capped by one of two large gabled projections from the hipped roof, and flanked by a pair decoratively-topped chimneys.

Brown, in whose honor it is named, founded the Portland Sugar Company in 1855, and in 1868 built the Falmouth Hotel, on Middle Street, which was a center of the city's social activities for half a century.