After the election of 1977 De Graaf was appointed as State Secretary for Social Affairs in the Van Agt–Wiegel cabinet, taking office on 28 December 1977.

After the election of 1986 De Graaf again returned as a Member of the House of Representatives, taking office on 3 June 1986.

De Graaf served as acting Minister of Social Affairs and Employment from 3 February 1987 until 6 May 1987 following the appointment of Jan de Koning as acting Minister of the Interior during a medical leave of absence of Kees van Dijk.

The Lubbers II cabinet fell on 3 May 1989 following a disagreement in the coalition about the increase of tariffs and excises and continued to serve in a demissionary capacity.

In September 1989 De Graaf was nominated as a Chairman of the Supervisory board of the National Health Care Institute, he resigned as State Secretary for Social Affairs and Employment the same day he was installed as chairman, serving from 1 October 1989 until 1 July 2003.