The property is today much reduced: its Federal period mansion was destroyed by fire in the 1930s, the hotel had a short-lived existence (1827-1859) before it was torn down on Gerrit Smith's orders, and numerous other outbuildings have also been demolished or lost to decay.

The wide piazzo on the southerly side of the house was converted into a large conservatory with a glass front and filled with choice flowers in great variety and heated in winter from a furnace in the basement of the mansion.

After dinner he conducted me over the grounds attached to his mansion consisting of 25 A[cres] of land through which was flowing a never-failing stream of water of sufficient size to operate a mill just below the premises.

The whole grounds were in a state of high cultivation, abounding in fruit and ornamental trees and flowers, producing abundance of vegetables in great variety, melons, etc.

The grounds were carefully laid out, with neat gravel walks passing in different directions with a design for convenience and ornament and they exhibited good taste.

Smith attempted unsuccessfully to make Peterboro a dry community, opening what is believed to be the first temperance hotel in the nation on the estate.

The estate was also widely known as a safe haven for escaped slaves making the trek to Canada on the Underground Railroad, and was a meeting place for suffragist organizations.