It is known for several prominent former residents, including: Thomas Oliver (1734–1815), royal Lieutenant Governor of Massachusetts; Elbridge Gerry (1744–1814), signer of the US Declaration of Independence, Vice President of the United States and eponym of the term "gerrymandering"; and James Russell Lowell (1819–1891), noted American writer, poet, and foreign diplomat.

Oliver owed his vast wealth to slave labor as both heir to Antiguan planters and absentee owner of Friar’s Hill plantation, Antigua, where 206 people were enslaved when inherited by his children.

[5] The Oliver-Vassall estate, located on the western edge of Cambridge, Massachusetts, included about 100 acres (40 ha) of rolling fields with a commanding view of the Charles River.

[6] The property extended from Fresh Pond in the north across the Charles River to what is now the Boston neighborhood of Brighton to the south, then part of Cambridge.

[9][10] On September 1, 1774, pursuant to orders given by Governor Thomas Gage, British Army troops removed provincial gunpowder from a magazine in what is today Somerville.

This activity caused a spontaneous rising of militia throughout the province amid rumors of actual violence that is known as the Powder Alarm.

[11] The next day Oliver was able to dissipate a crowd that formed in Cambridge (near present-day Harvard Square) by going to Boston, conferring with Gage, and reporting that no further military movements were planned.

[6] Early in the siege of Boston that began after the battles of Lexington and Concord in April 1775, the house was occupied by troops that eventually became part of the Continental Army; one of the building's notable occupants during this time was Benedict Arnold, then in the Connecticut militia.

[14] Gerry served as Governor of Massachusetts in 1810 and 1811; redistricting of the state in 1812 prompted the coining of the term "gerrymandering" to describe the practice of shaping legislative districts in partisan ways.

He had difficulty coping and became a recluse at Elmwood for a time[24] until an invitation to speak at the Lowell Institute resulted in a job offer at Harvard College.

Instead, he took lodging in an area known as Professors' Row on Kirkland Street in Cambridge along with his daughter Mabel and her governess Frances Dunlap.

[31] Lowell remained at Elmwood for the remainder of his life with a few exceptions, including the period between 1877 and 1885 when he served as Minister to Spain and Great Britain.

At Elmwood, he wrote some of his best-known works, including The Vision of Sir Launfal, The Biglow Papers, and A Fable for Critics, all published in 1848.

He mentions the home in some of his poetry:[33] My Elmwood chimneys seem crooning to me, As of old in their moody, minor key, And out of the past the hoarse wind blows.

Lowell's friend and fellow poet Henry Wadsworth Longfellow also wrote a poem about the house called "The Herons of Elmwood".

[34] Beginning in the summer of 1872, when Lowell traveled to Europe, he rented the house to Thomas Bailey Aldrich and his family.

[32] After the death of James Russell Lowell the house was inherited and used seasonally first by his daughter Mabel, who had by then married Edward Burnett, then by their children.

Porter, a Harvard professor, used the house as a private residence, but also taught some of his classes there and allowed students to use his extensive library.

Acting President Derek Bok moved his family there in 1971 amid security concerns originating in student protest activity near the then-president's residence on Quincy Street.

It was at first administered by the Metropolitan District Commission as part of the Charles River Reservation; the MDC's successor, the Massachusetts Department of Conservation and Recreation, is now responsible for the park.