[2] He had a few scholarship offers in his senior year at Northwestern High School in Hyattsville, Maryland, but he didn't have the grades that were good enough to attend college.

He was working at a local JC Penney's and playing semi-pro football in the Washington, DC-Virginia-Maryland area for the DC Explosion (NAFL), when he was noticed by a coach who worked at Saint Paul's College, Virginia.

Toler made the Central Intercollegiate Athletic Association (CIAA) All-Conference team as a junior and senior.

Considered one of the most intriguing small-school prospects in the 2009 NFL draft, Toler was graded as the seventh-best cornerback available.

[6] Toler was placed on injured reserve on January 6, 2014, after suffering a groin injury during the Colts 45–44 Wild Card win over the Kansas City Chiefs.