During the late '50s, Lytle continued to box, but landed jobs as a drummer for Ray Charles, Jimmy Witherspoon and Gene Ammons.
He formed his first group in 1957 with saxophonist Boots Johnson, organist Milton Harris and drummer William "Peppy" Hinnant.
Throughout his career he performed and recorded with jazz greats including Louis Armstrong, Lionel Hampton, Miles Davis, Nancy Wilson, Bobby Timmons and Roy Ayers.
Lytle remained a popular concert attraction in the U.S. and Europe; his last performance was with the Springfield (Ohio) Symphony Orchestra in his hometown on November 18, 1995.
Passing away from kidney failure the following month, he was survived by his wife Barbara Jean Lytle, his son Marcel Anthony (of Atlanta), Michael-Lamont (of Toronto), and daughter Ayo Michelle Hagans (of Springfield).