When he was six months old, his family moved to Philadelphia, Pennsylvania, due to threats his father received from local white supremacists.

[2] Smith spent the remainder of his childhood in Philadelphia, graduating from Benjamin Franklin High School.

Smith became a part of jazz history as the bassist on Coleman Hawkins seminal 1939 recording of "Body and Soul".

During World War II Smith acted as the band director in the Thirty-Seventh Special Services Company, U.S. Army, stationed in Fort Huachucha, Arizona.

During this period, Smith began his lifelong career as an educator, teaching at Seward Park High School in New York.

In 1962, Smith became the second black member in the history of the Nashville Symphony Orchestra, with which he played double bass and viola for seventeen years.

Smith also worked as an adjunct professor at the Blair School of Music at Vanderbilt University.