Irby was born and raised in Tuscaloosa, Alabama, and found his calling to music at the age of 12.
During that tenure, he also recorded and toured with Marcus Roberts and was part of Betty Carter's Jazz Ahead Program together with Roy Hargrove.
After a four-year stint with Hargrove, Irby focused on his own group, in addition to being a member of Elvin Jones' ensemble and Papo Vazquez's Pirates Troubadours.
From 2003 to 2011, Irby was a regional director for JazzMasters Workshop, a mentoring program for young children.
Since rejoining, Irby, along with most members of the orchestra, has arranged much of the vast library of music that they have performed.