Earlier in his career, he served one term in the Wisconsin State Assembly (1949), representing Brown County as a Democrat.
Born in North Branch, Minnesota, Duffy grew up in Hollandtown, Wisconsin, and graduated from Kaukauna High School.
He graduated from St. Norbert College and received his Juris Doctor degree from the University of Wisconsin Law School in 1941.
Later that year, he joined the United States Army Air Forces for service in World War II.
In 1948 he defeated incumbent Republican Assemblyman Harvey Larsen to serve in the Wisconsin State Assembly for the 1949-1950 session.