A native of New Orleans, Richardson attended L. B. Landry High School in Algiers as a freshman on the junior varsity team.

[1] They moved to the suburbs of Baton Rouge, Louisiana, where Richardson attended Baker High School for his sophomore year.

A late-season addition to the Baker football team, he rarely played before the family moved to Fort Worth, Texas, in October 2006.

[1] In Fort Worth, Richardson enrolled at North Crowley High School, where he joined the football varsity but did not play during his junior season for academic reasons.

As a sophomore in 2011, Richardson replaced Watkins at left tackle, starting all 13 games of the season to protect quarterback Robert Griffin III's blindside.

He earned mid-season All-American honors by ESPN as the only selection from an unbeaten Baylor Bears team.

[4][5] Daniel Jeremiah of NFL.com named him one of college football's “most physical players.”[6] Richardson was selected by the Buffalo Bills in the fifth round, 153rd overall.