In 1940 he moved to New York City, and in the first half of the decade played in the bands of Fats Waller, Claude Hopkins, Edgar Hayes, Ella Fitzgerald, Sabby Lewis, and Machito.
From 1945 to 1949, and again in 1951, he played and recorded extensively as a member of Duke Ellington's orchestra.
Williams worked primarily with Latin jazz ensembles and New York theater bands in the 1950s and 1960s, and played with Clyde Bernhardt and the Harlem Blues and Jazz Band in addition to working with his own quartet.
Williams was a single father and had one son, actor Greg Morris.
[2] He was survived by his son Greg Morris and his niece Florence Randolph.