De Graaf worked as a researcher at the Radboud University Nijmegen and the Centrum voor Parlementaire Geschiedenis [nl] from July 1981 until September 1985.

He sat as vice-chairman in the parliamentary inquiry commission that looked into the investigative methods used by the Dutch inter-regional police force, leading to the resignation in 1994 of the Minister for Internal Affairs, Ed van Thijn.

After the Parliamentary leader of the Democrats 66 in the House of Representatives Gerrit Jan Wolffensperger [arz; nl] announced that he was stepping down as Parliamentary leader in the House of Representatives following increasing criticism on his leadership, the Democrats 66 leadership approached De Graaf as his successor, De Graaf accepted and became the Parliamentary leader in the House of Representatives, taking office on 21 November 1997.

In 2005, de Graaf resigned, after the introduction of democratically elected mayors had been rejected in the Senate, with a deciding vote cast by the Labour Party faction under guidance by Ed van Thijn.

On 23 March 2005, De Graaf resigned after a proposed constitutional reform on elected-mayors was rejected by the Senate.

De Graaf semi-retired from active politics and became active in the public sector and occupied numerous seats as a nonprofit director on several supervisory boards (Centrum voor Parlementaire Geschiedenis, Consumentenbond and the Anne Vondeling prize) and served on several state commissions and councils on behalf of the government (Public Pension Funds APB, De Koning Commission, National Committee for 4 and 5 May, Netherlands Film Fund and the Advisory Council for Spatial Planning).

De Graaf was elected as a Member of the Senate after the Senate election of 2011, taking office on 7 June 2011 serving as a frontbencher chairing the parliamentary committee for Kingdom Relations and spokesperson for the Interior, Kingdom Relations, European Affairs, Defence and Immigration and Asylum Affairs.