Charles Carroll of Annapolis inherited and extended his father's fortune but, as a Roman Catholic, was barred from participation in Maryland politics.

Charles Carroll the Settler came to the colony in 1688 with a commission as Attorney General from the colony's Catholic proprietor, Charles Calvert, 3rd Baron Baltimore, but after only a year lost that position as a result of the Province's so-called "Protestant Revolution", a rebellion of Protestant settlers associated with the Glorious Revolution in England.

The royal government that took over the Colony, after moving the founding capital from the Catholic stronghold of St. Mary's City on the shores of the Potomac and Chesapeake in southern Maryland to the more central and renamed Annapolis near Kent Island in 1694; banned Catholics from holding office, bearing arms, serving on juries, and eventually from voting.

Barred therefore from a political career, Carroll the Settler turned his attention to business, amassing a fortune so large that he was eventually the wealthiest man in Maryland at the time of his death in 1720.

His son Charles Carroll of Carrollton, (1737–1832), eventually secured his family's vision of personal, political and religious freedoms for all citizens when he became the only Roman Catholic (and last surviving signer) to sign the Declaration of Independence in 1776, while becoming the richest man in America [citation needed] and having a profound effect on his nation, state and city.