In 1862, the U.S. War Department built various temporary wooden barracks-style buildings for the Jarvis Military Hospital on the grounds, to care for wounded Union soldiers.

He chose to live at "Mount Steuart", his large family plantation further to the southeast of the city of Annapolis on the South River in Anne Arundel County.

The modern Grace Medical Center, was constructed on the site in 1919 by the religious order and is operating today.

[3] The mansion was located at the present-day junction of West Baltimore and Monroe streets and built on relatively high ground, at the time on the western outskirts of the city.

[3] On May 8, 1829, James Steuart's daughter Elizabeth was married to the writer and essayist George Henry Calvert, at Maryland Square.

[4] On July 19, 1844, the Boston City Greys of the Massachusetts state militia visited Baltimore, and marched in parade with various companies of the 53rd Regiment.

After the Civil War, the Maryland National Guard was organized as a type of successor to such local militias.

The grounds were used for temporary, numerous wooden barracks-type buildings constructed for the care of wounded Union soldiers.

[14] In February 1862, a Massachusetts soldier described the property (by then known as "Camp Andrew", after Massachusetts Governor John Andrew): We are nicely quartered on a high hill situated on the west of Baltimore formerly owned by Gen. Stewart [sic] now of the Rebel Army and the property is now confiscated.

He lived at "Mount Steuart", the large family plantation on the Chesapeake Bay in Anne Arundel County.

In 1919, the Sisters of Bon Secours constructed and opened a hospital on the site, their first in the United States, at 2000 West Baltimore Street.

[17] The Grace Medical Center continues today as a part of the modern neighborhood in old West Baltimore, which retains the name of "Steuart Hill".