He entered the counting-house of his brother Samuel, a Philadelphia merchant, but he spent much of his time in outdoor sports, and until 1818 was president of the famous fox-hunting club of Gloucester, New Jersey that was organized in 1766.

In 1796 he was made alderman of that city, and in the same year quelled a riot among sailors who had organized themselves into a body and demanded higher wages.

After reading the riot act, he requested they disperse, and, being received with shouts of defiance, Wharton ordered each of his men "to take his man," and the sailors were captured and imprisoned.

His 14 elections to the office make him the longest-serving and most-elected mayor in Philadelphia's history.

He was buried at Friends Western Burial Ground at 16th and Race Street in Philadelphia.