Born in a log cabin in the backwoods of northwest Georgia, as a young man he trained horses for the United States Cavalry and worked on a cattle ranch.

On November 8, 1945, Smith was suspended from racing for a year by The Jockey Club after being found responsible for administering the stimulant ephedrine via an atomizer to one of his horses.

In his absence, Roy Waldron trained for a time for Maine Chance Farm, winning the Pimlico Futurity with Star Pilot, before Smith's 36-year-old son, Jimmy, took over for the remainder of the suspension.

When his suspension was over, Smith returned to Maine Chance Farm, where he trained 1947 Kentucky Derby winner Jet Pilot.

He died two years later in Glendale, California, and was buried there in the Forest Lawn Memorial Park in Sunrise Slope, Lot 6121, Space 4.