Prosser joined Tottenham Hotspur at the age of ten after being scouted playing in a soccer school run by Micky Hazard.
[9] Following an injury to Gareth Owen in March 2009, Prosser was allowed to affirm his place in the team and hoped to earn a new contract at the end of the 2008–09 season.
[11] However, two months later he was transfer listed by new manager Micky Adams in August 2009 along with five other young players, having failed to impress in pre-season.
[12] He swiftly joined Conference Premier club Salisbury City on a one-month loan deal along with teammate Danny Glover.
[17] Another injury to Gareth Owen allowed Prosser to return to the first XI for a league clash with Rotherham United on 14 November 2009.
[25] Prosser made 27 appearances for Southend in all competitions during the 2011–12 season,[28] as the club narrowly missed out on automatic promotion and failing in the play-offs.
[38] Prosser was named in the Football League Team of the Week for his performance in a 1–0 win over former club Port Vale at Roots Hall on 10 October 2015.
[45] He made his competitive debut for the club in Colchester's 1–1 draw with Hartlepool United at Victoria Park on the opening day of the season.
[46] After making 17 appearances for the Colchester United first-team, Prosser was ruled out for the remainder of the 2016–17 season in November 2016 following surgery on his knee.
[54] He made 41 appearances during the 2018–19 campaign and was named on the EFL Team of the Week for his performance in a 1–0 win over champions-elect Lincoln City on 27 October 2018.
[55] He played in United's 0–0 draw with, and subsequent penalty shoot-out victory over, Tottenham Hotspur in the EFL Cup on 24 September 2019.
[58] However, Colchester were in sixth-place when the season was suspended due to the COVID-19 pandemic in England, which led Prosser to comment that "there's so much to play for and we're desperate to get back out there with the boys".
[59] Chairman Robbie Cowling went on to confirm that Prosser would be offered a new contract by the end of June, admitting that he was one of a number of players that "under normal circumstances I would have been determined to re-sign".
[63] Manager Alex Revell said that Prosser would have "a huge responsibility every day to not only lead the squad with his experience but show the younger players what it takes to be a top professional".
[71] He later joked that "It's funny when the chauffeur pulls up and your sister gets in and goes off with England and I'm sitting there thinking: 'Bloody hell, I'm at Port Vale here, where's my life going?