His grandmother, Mercy Harris-Hampton, was the daughter of Benjamin Harris, the "fighting Quaker" of the American Revolutionary War.

[1] The elder Brewster was a successful and well-known attorney in Philadelphia who had abandoned Benjamin's mother, Maria Hampton, for her companion Isabella Anderson, by whom he had two children out of wedlock.

Benjamin's sister, Anne Hampton Brewster (1818–1892), was one of America's first female foreign correspondents, publishing primarily in Philadelphia, New York and Boston newspapers.

[3] In 1857, he married as his first wife, Elizabeth von Myerbach de Reinfeldts, the widow of Dr. Shulte of Paris, France.

She was the daughter of Robert John Walker, Secretary of the Treasury under President James Knox Polk[4] and Mary Blechenden Bache.

Sophia was the daughter of Arabella Maria Smith and Alexander J. Dallas an American statesman who served as the U.S. Treasury Secretary under President James Madison.

In addition she was a niece of George Mifflin Dallas, the 11th Vice President of the United States, who served under James K. Polk.

[9][10] In 1846 Brewster was appointed commissioner by President James K. Polk to adjudicate the claims of the Cherokee against the U.S. federal government.