Electing to forgo a career in law, Patterson studied in a banking house and founded a series of mills in Salem, North Carolina.

[1] He was the eldest son of Samuel F. Patterson, a politician who was a North Carolina State Treasurer, and Phoebe Caroline Jones.

[3] His maternal grandfather was North Carolina politician Edmund Jones and his great-grandfather was Revolutionary War officer William Lenoir.

[4] Patterson split time in his youth at Caldwell County and Raleigh, North Carolina, where his father worked.

[5] Patterson graduated from the University of North Carolina in 1851, then studied law under future U.S. Representative John Adams Gilmer.

However, he found the study of law unappealing and, after a brief period farming at the family homestead, decided to pursue a career in business.

Although he was disillusioned with the direction the party was heading, he nonetheless approved the state's ordinance of secession at the 1861 North Carolina Constitutional Convention.