It is a 2+1⁄2-story wood-frame structure, five bays wide, with a side-gable roof, a single off-center interior chimney, and clapboard siding.

The (east-facing) front is symmetrical, with a central entrance flanked by short sidelight windows and pilasters, and topped by a Federal style semi-oval fan.

[2] The house was built in 1797 by Isaac Gilkey, a local carpenter who is credited with other buildings in the South Street area.

James Phinney Baxter repurchased his birthplace in 1907, and donated it to the town, with the stipulation that a library be built on its site, and that the home be preserved as a museum in honor of his father.

[2][3] James Phinney Baxter served six terms as Mayor of Portland, Maine (1893-1896 and 1904–1905) and was a leading local businessman and philanthropist.