Having witnessed the defeat and dissolution of the Austro-Hungarian monarchy, he entered politics and, through the mediation of the deputy Otto Glöckel, found employment as the secretary of the Social Democratic president of the National Council parliament Karl Seitz.

Schärf, as well as Karl Seitz and the Austromarxist party official Otto Bauer, had urged Renner to step down from office, which proved to be fatal as it gave the government of Chancellor Engelbert Dollfuss the opportunity to overthrow the parliamentary system.

Schärf became a deputy of the Federal Council in 1933, and though he kept aside from the Social Democratic Republikanischer Schutzbund paramilitaries, he was arrested after the 1934 February Uprising and lost his public offices in the course of the establishment of the Austrofascist dictatorship.

Immediately after the Soviet Vienna Offensive and the occupation of the city in April 1945, Schärf became acting chairman of the refounded Social Democratic Party of Austria and joined the Austrian national unity government of Chancellor Karl Renner.

He served as Vice-Chancellor in the grand coalition governments between the Conservative People's Party and Social Democrats (the Communists were pushed out in 1947) under Chancellor Leopold Figl and his successor Julius Raab until 1957.

In 1955, Schärf together with Chancellor Raab and Foreign minister Leopold Figl took part in the Moscow negotiations for the Austrian State Treaty, whereby he expressed strong reservations against the Declaration of Neutrality.

A firm supporter of the Austrian Proporz system and collaborating with three Conservative chancellors (Raab, Gorbach and Klaus), Schärf gained recognition by exercising his office according to the principle of non-partisanship.