Born and raised in the Baltimore area, he was the younger brother of Arthur Pue Gorman (1839-1906), an influential political leader and longtime United States Senator from Maryland.

[8][verification needed] In 1891, Gorman and his brother Arthur founded the Cumberland Coal Company in Tucker County, West Virginia with 300 employees and 140 coke ovens.

[9] Gorman and his father Peter were the proprietors of several quarries in the Laurel area and adjacent Prince George's County, Maryland (northeast and east of Washington) that supplied granite and stone for the U.S. Treasury Department building (east of the White House) and the 1850s building expansion project of the new House of Representatives and the Senate wings of the United States Capitol, in Washington, D.C., along with material for bridge construction on the Baltimore and Ohio Railroad, first passenger railway in the country, begun 1827-1828.

[13] Gorman married Ada Rogers, widow of Judge Edward Thomas Boykin, in a private ceremony on June 18, 1903, in Concord, North Carolina.

[14] Gorman purchased property in the streetcar suburban village of Catonsville, Maryland's Oak Forest Park neighborhood (of the then rural surrounding Baltimore County) in 1896 for $2500 and contracted with William Gerwig in 1897 to build a country residence there.

[15][16] Gorman's daughters, Nora and Elizabeth, both announced their engagements in September 1903, to marry E.G. Ballenger of Tryon, North Carolina, and J. William McMillan of Knoxville, Tennessee, respectively.

[24][25] After two weeks in Atlantic City, New Jersey, where they had previously visited,[26] his third wife and widow Ada Rogers Boykin Gorman returned to her former Concord, North Carolina home in August 1915.