[5] He had spent ten years as a trainee for the London club, working under Steve Clarke and training alongside the likes of John Terry, Frank Lampard and Eiður Guðjohnsen, with Carlton Cole and Robert Huth in the same youth team as Baldwin.
[13] In January 2008, Baldwin suffered shoulder and knee injuries that would keep him out for nine months, unable to help Colchester's relegation from the second to the third tier of English football.
[14] In January 2009, Baldwin was involved in the U's first away clean-sheet since August 2007 in a 0–0 draw with Bristol Rovers, a span of 17 months, having played in both fixtures, the other being a 3–0 away win at Preston North End.
[17] Following an injury lay-off and falling out of favour under new Colchester manager Aidy Boothroyd, Baldwin was signed by Bristol Rovers on loan in November 2009.
[18] He made six appearances on loan with Bristol Rovers[19] but his future at Colchester was in doubt, with manager Boothroyd telling Baldwin that he was fifth in line to Magnus Okuonghae, Paul Reid, Danny Batth and Matt Heath in central defence.
[20] He was set to sign for Bristol permanently following his loan spell in early January 2010,[21] however with a transfer fee agreed between the two clubs, Baldwin decided to stay and fight for a place with Colchester.
[22] On 29 January 2010, both Baldwin and teammate Scott Vernon signed on-loan for rivals Southend United until the end of the season, but neither were able to play in the Essex derby on 8 February.