Built in 1804 for Captain Thaddeus Tuttle (1758–1836), a local merchant, the building was designed by architect and surveyor John Johnson and constructed by carpenter Abram Stevens.

Allen claimed that he had been swindled out of the land by Captain Thaddeus Tuttle while he was abroad attempting to negotiate trade with England, and to purchase arms for the Vermont militia from France, at the behest of Governor Thomas Chittenden.

By the time of Allen's return his financial empire was in shambles, he had lost most of his land holdings, and was eventually forced to flee Burlington or face debtors' prison.

By 1804, the same year that he had Grasse Mount built, he had also constructed a store on the corner of Main and South Prospect Streets in Burlington, Vermont, at the building known today as "Bittersweet".

Historical accounts reflect the reason for Tuttle's bankruptcy was due to his business operating primarily on the basis of barter where there was very little handling of actual monies, and therefore the construction and sustained maintenance of Grasse Mount "was more than he could carry".

[9] In 1826, Van Ness declined re-election to the Governorship and practiced law in Burlington until he accepted an appointment of United States Ambassador to Spain by President Andrew Jackson in 1829; a position he held until 1839.

[5] On July 29, 1825 during his tour of the twenty-four states of the country, the famed American Revolutionary War General, Marquis de Lafayette was entertained at the Grasse Mount estate after laying the cornerstone for the "South College" building at the University of Vermont.

The remainder of the estate including the mansion was sold in March 1853 for $12,000 to Captain Charles B. Marvin, a retired naval officer and merchant who had survived the shipwreck of a vessel which he commanded in 1848, and then made a fortune during the 1849 California Gold Rush.

Spending about $10,000 over the course of a decade, Marvin employed what historians estimate to have been at least two Italian professional artisans (most likely a master and an apprentice) who used watercolors and distempers to hand-paint numerous scenes from his seafaring years, including palm trees, shorelines, windmills, ships, international seaports, as well as cherubs, garlands, and other classical patterns in the form of trompe-l'œil (translated in French as "trick of the eye"), where the images convey the optical illusion of having three-dimensions.

On 7 April 1866, Lawrence Barnes (1815-1886) of Hillsboro, New Hampshire, a local businessman involved in the lumber industry who served as president of the Howard Bank,[15] and was noted for helping to introduce the marble trade in the city (as well as posthumously, becoming the namesake of the Lawrence Barnes Elementary School in Burlington) purchased the estate for $35,000 from Captain Marvin who decided to move with his family back to California during that year.

[4][16] By this time, the perimeter of the estate was greatly reduced, bounded by six parcels along Willard St. to the west, by Summit St. to the east, Main St. to the north, and Maple St. (née Prospect St.) to the south.

[16] On 19 May 1894, Edward Wells purchased the property from the Barnes family estate in order to buffer his home (which abutted Grasse Mount to the south) from potentially undesirable neighbors.

In 1984, workers renovating Grasse Mount for the relocation of the alumni and development offices had unexpectedly discovered unique nineteenth-century wall art, which had been hidden under layers of paint and wallpaper.

Eric Groves, an architectural conservator who was working for the firm Kielman and Batten had first discovered stenciled wall and ceiling imagery on the second floor southeast chamber, which led to an extensive investigation of each room in the building.

The university architect at the time, Robert Holdridge contacted the National Park Service, which referred the restoration team to the Albert K. Perry Company of Boston, Massachusetts.

[26] The three-dimensional trompe-l'œil imagery within the top-level cupola lantern simulates columns and capitals studded with green jewels that project the illusion of sunlight emanating through the windows.

However, some of the rooms were restored with a "window" view to the artwork, the southeast parlor was left uncovered, and two decorated ceilings were removed to storage at the university's Fleming Museum of Art.

The remainder of the revealed artwork was surveyed, documented (by taking photographs and tracings), and then covered with a protective non-vinyl wallpaper, affixed with a water-based adhesive for ease of removal during future restoration work.

[30] Considered to be a sophisticated example of federal domestic architecture,[31] Grasse Mount is a two-story brick mansion with a hipped-roof, cupola, and balustrade which runs along the perimeter of the roof adorned with cornices between the second story window bays, which are flanked by exterior louvered shutters.

A brick belt course divides the first and second floors, where six Ionic pilasters originate between each bay and extend to the rooftop cornices, and summit at small urns atop the balustrade.