Williams graduated from Yale University in 1958 with a Bachelor of Arts, magna cum laude, where he was a member of the Manuscript Society.

He then attended Harvard Law School, where he was classmates with future federal judges Anthony Kennedy, Laurence Silberman, A. Wallace Tashima, R. Lanier Anderson III, and Timothy B. Dyk.

He was in private practice at the law firm Debevoise & Plimpton from 1962 to 1966, and from 1966 to 1969 he was an Assistant United States Attorney for the Southern District of New York.

Williams was nominated by President Ronald Reagan on February 19, 1986, to a seat on the United States Court of Appeals for the District of Columbia Circuit vacated by Judge Malcolm R. Wilkey.

[1] In March 2017, Williams questioned if the government could constitutionally keep all prisoner court filings secret when the court, unanimous in judgment but in divided opinions, found that the press could not access classified video of Jihad Ahmed Mustafa Dhiab being force fed during the Guantanamo Bay hunger strikes.