Born and raised in Waterloo, a populated place in Juniata County, Pennsylvania, he attended public schools and then Jefferson College in Canonsburg.

He first entered politics in 1859 when he was elected to the Pennsylvania House of Representatives, in which he served until 1861 when the Civil War began.

He joined the United States Army and served as a captain in the Fifteenth U.S. (regular) Infantry.

After the war Patterson moved to Columbia, South Carolina, and engaged in railroad construction.

He was known for speaking out against events such as the Hamburg Massacre and supporting the rights of African-Americans in South Carolina.