At Lawrence Free State High School Ballard competed in football, basketball as well as track.

Prior to his senior season was listed amongst the Rivals.com Top 100 and ranked the #4 tight end prospect in the nation by Sports Illustrated.

As a junior, he made a verbal commitment to then Kansas football Coach Mark Mangino, but continued to receive interest from other schools.

After receiving notice via the internet that Ballard took a recruiting trip to the Georgia[2] it is rumored that Mangino rescinded the Jayhawks scholarship offer.

At the University of Iowa Ballard made an immediate impact by playing as a true freshman and collecting 15 tackles, four tackles-for-loss and 2.5 quarterback sacks from the defensive end position.

However, soon after, it was reported that he and another player, Justin Houston of the University of Georgia, had tested positive for marijuana.

On August 19, 2013, coach Leslie Frazier announced that Ballard would be taking time off from the team to deal with personal issues.

[6] The next month, in an interview with USA Today, Ballard stated: "I wasn't really having a good time playing football.