Brown spent summers at Cushing Academy, a private boarding school in Massachusetts, with his sister Candice.

With no prospects for a college scholarship in football, Brown attended the Milford Academy in New Berlin, New York,[3] where NFL starters LeSean McCoy and Shonn Greene had played.

[4] Brown played college football for the Temple Owls, where he was a punt and kickoff return specialist, wide receiver and, for his first three years, back-up running back to Bernard Pierce.

[1][5] Because of his size, 5 foot, 5 inches tall,[4][6] no Division I or II schools showed any interest in Brown; even at Temple he was a walk-on and played his first year without a scholarship.

[8] Brown attended both the Tampa Bay Buccaneers and the Kansas City Chiefs free agent camps in the spring of 2013,[7] but did not hear back from either.

Two weeks later, he was preparing to sign a contract with the Saskatchewan Roughriders of the Canadian Football League, but was denied entry to Canada due to an expired passport.