[1] Baldwin was born in 1834 at his family's estate, Bunker Hill, near Millersville in Anne Arundel County, Maryland.

[2] His father served with distinction in the War of 1812, under Commodore Lewis Warrington, and his grandfathers both served in the American Revolutionary War, commissioned officers in the 1st Maryland Regiment under General William Smallwood.

[4][5] In this role, he served as president of the Louisville and Nashville Railroad for four years when he resigned due to management differences with the other major stockholders.

[7][8] He owned a cottage in Newport, known as the "Baldwin villa" which was located at the corner of Bellevue Avenue and Narragansett.

[13][14] She was the daughter, and sole heir, of James Dixon Roman (1809–1867), a U.S. Representative who was president and part owner of Old Hagerstown Bank.