He is the son of Milan D. Smith, Sr., who served as chief of staff of the Department of Agriculture under Secretary Ezra Taft Benson.

Smith attended the University of Chicago Law School on a full-tuition scholarship, graduating in 1969 with a Juris Doctor.

After law school, Smith became an associate attorney at the Los Angeles firm of O'Melveny & Myers.

Smith was nominated by President George W. Bush on February 14, 2006, to fill a seat vacated by Judge A. Wallace Tashima.

[6] In 2022, Smith told the Deseret News that he has no plans to retire and wishes to "die with my boots on.