Born on the Army base at Fort Leavenworth, Kansas, Smith began his music education very early organizing bands in Wichita starting in 1893.

His strict military style leadership led to prominence and over the next 30 years he would lead bands in Chicago, Wichita, Kansas City, the Tuskegee Institute, and in St. Louis.

[2] However, eye problems prevented him from pursuing a military career, so he moved with his wife, Laura Smith (née Lawson), to Wichita, Kansas in 1893.

During this time, he went with the infantry unit to the front lines in the Spanish–American War where he met Theodore "Teddy" Roosevelt at the Battle of San Juan Hill.

[2] After eight years at the Tuskegee Institute, Smith decided to leave because of a rumored disagreement between him and Washington about the number of plantation songs in his orchestral arrangements.

[3] Once again, his connection with Teddy Roosevelt led him to accept a position to head the military and music departments at the Western Baptist University in Kansas City in 1913.

After attending the Joe Louis versus King Levinsky boxing match in Chicago in August 1935 and returning home to Kansas City, he suffered a stroke.

[2] The University of Missouri-Kansas City page about Smith calls him "America's Greatest Colored Bandmaster" and his legacy was significant on African-American music and particularly on jazz.

One student, Harlan Leonard explained that Smith was "short, gruff, military in bearing, wore glasses and was never without his full uniform and decorations.