To this is attached a larger 1+1⁄2-story square brick building dating to 1964, when the museum and library were organized.

This building, composed of three separate 19th century structures later joined, occupies a site said by tradition to be the law office of future United States President James Monroe from 1786 to 1789.

Monroe had prior to this time served in the Virginia House of Delegates and in the Congress of the Confederation, from which he resigned in 1786.

He accumulated a variety of artifacts related to his illustrious ancestor, restored the property, and opened it as a museum.

It is now managed by a board consisting of members of the Monroe Foundation and the regents of the University of Mary Washington.