[2] Brown was selected by the Phillies in the 20th round of the 2006 Major League Baseball Draft out of Redan High School in Stone Mountain, Georgia.

[3] Since he moved to the Atlanta area from Pasco High School in Dade City, Florida, Brown had "fallen off some teams' radars" prior to the draft.

[4] He participated in a private batting session with Phillies representatives at a local park, and after making one adjustment to his swing, scouting director Marti Wolever "couldn't believe what [they] were seeing".

[4] Brown planned to attend the University of Miami to play wide receiver for the Hurricanes, but the Phillies offered him a $200,000 signing bonus to choose baseball instead.

[6] His performance allowed him a late promotion to the advanced-A Clearwater Threshers, where he batted .444 with one home run and seven RBI in eleven plate appearances.

[10] After promotion to the Double-A Reading Phillies, Brown batted .279 with 20 RBI and 8 stolen bases; he also hit three home runs and four triples (tied for the team lead).

[13] In a game against the Tigers, he hit two home runs—one against Justin Verlander[14]—and added a bases-loaded infield single to collect a third RBI.

"[4] First baseman Ryan Howard nicknamed Brown the "Total Package", noting that "I've seen him take batting practice" and that "[he's] got pop to all fields".

[14] Though it was speculated that he would begin the year with the Triple-A Lehigh Valley IronPigs,[15] he was assigned to Reading for the start of the 2010 season, where he batted .333 in his first seven games.

[9] He drove in one of two runs for the Double-A club on an RBI triple to back Phillippe Aumont's no-hit bid on April 25.

Chuck LaMar, the Phillies' assistant general manager, said that Brown "is just now starting to scratch the surface on his ceiling as a potential Major League player", praising his hand–eye coordination and his power.

[20] He started for the U.S. team, but left in the first inning after experiencing tightness in his hamstring muscle running from the batter's box on an infield single.

He batted in Raúl Ibañez with a sacrifice fly in the bottom of the seventh inning for his second career RBI, finishing with two hits in three official at-bats.

Brown received a standing ovation from the crowd prior to his first plate appearance; he later said that the experience "was great" and that he "wasn't thinking about that at all".

[25][26] He was sent down to the Lehigh Valley IronPigs, the Phillies Triple-A minor league affiliate, on July 29, 2011, to open a roster spot for Hunter Pence.

Ryan Dinger of Phillies Nation commented that Brown "showed flashes of being the player everyone thinks he can be", but that "he was also plagued by long stretches of ineffectiveness".

Sporting News rated Brown the worst defensive left fielder in the major leagues after three blatant blunders during June.

[27] On February 25, 2016, Brown signed a minor league contract with the Toronto Blue Jays that included an invitation to spring training.