Born to the prominent Wadsworth family in New Hartford, Connecticut, he attended private academies until he was sixteen, when he entered the dry goods trade.
In 1853, he moved to Chicago, Illinois, to join his brother in his dry goods operation, eventually rising to become president of Philip Wadsworth & Co., a clothing store.
The Wadsworths were a prominent family in Connecticut; his great uncle Jeremiah was a government official for the Continental Army and his father Tertius was a wealthy real estate developer.
Philip Wadsworth was raised in New Hartford and attended public schools before studying at Williston Seminary and the Connecticut Baptist Literary Institute.
Wadsworth was offered a regiment with the rank of colonel by Governor Richard Yates, but he declined, citing the need to maintain his business interests.