The gatehouse was originally a two-story rough stone farm house built by Richard Whittier between August and November 1830.

It was redesigned in 1883 by architects Damon Brothers into a gatehouse for the 76-acre (310,000 m2) Tenney estate known as Grey Court.

The modifications by Damon gave the building a distinctive Queen Anne Victorian character, changing the roof to be hipped, and adding an ornately decorated tower with weathervane on one corner.

In the 1970s, after the Basilian Salvatorian Order vacated the property, the castle was used as a substance abuse treatment center and was then abandoned, boarded-up, looted and vandalized.

Instead of preserving the still-impressive ruins of this Carrere and Hastings Beaux-Arts mansion, which despite the fire, still featured original exterior terracotta details, brickwork, and granite walls, the remains were almost completely demolished by approval of the Massachusetts Department of Environmental Management.