He was the 39th Lieutenant Governor of Wisconsin (1979–1983) and was afterward appointed Midwest Director for the United States Department of Transportation in the administration of U.S. President Ronald Reagan.

He served in the United States Marine Corps from 1942 to 1946, assigned to the Pacific theater of World War II.

He moved to the rural town of Randall in Kenosha County, Wisconsin, where he raised cattle and started a family.

In May 1964, the Racine County judge who had been assigned the case, John Ahlgrimm, ruled in favor of Elfers.

Olson later served on the UW Medical Center board and as a member of the Wisconsin Building Commission.

Olson broke with mainstream Republicans in opposing the tax cuts of the 1980s, believing that doing so would make budgets unsustainable.