Lee played violin in the pit and performed the oboe onstage in one scene; he was one of only two African-American musicians in the orchestra.

[5] When Carmen Jones's conductor Joseph Littau fell ill, Lee "got his first break as an emergency pinch hitter".

[9] Leonard Bernstein saw a performance of Carmen Jones with Lee leading the orchestra and asked him to become the permanent conductor of his musical On the Town.

In 1947, he founded an interracial orchestra, the Cosmopolitan Symphony Society, made up of "Americans of Chinese, Russian, Jewish, Negro, Italian and Slavic origin", as well as women.

Oscar Hammerstein II declined to hire Lee to conduct touring productions of his shows, explaining that Southern theaters would refuse to book them.

Featured soloists included, Benjamin Matthews, baritone (co-founder of Opera Ebony), Joy Simpson, Alpha Floyd, Joyce Mathis, sopranos and Michael Austin, tenor.

In 1976, he conducted the New York Philharmonic for the first time; the concert was in honor of Martin Luther King Jr.'s birthday and included a work by African-American composer David Baker.