On 29 October 1998 he joined Torquay United, initially on a one-month contract to cover for a central defensive injury crisis.
He made his debut for Torquay in the 1–1 draw with Scarborough two days later and within that initial month played well enough to earn a permanent contract, though struggled to fully establish himself with Wayne Thomas, Lee Russell and Alex Watson all ahead of him.
After being sent off for violent conduct against Mickey Adams' flying Brighton & Hove Albion side, Aggrey returned that season with some outstanding and commanding performances.
He finally left Plainmoor on 10 December 2001, after agreeing a settlement on the remaining six months of his contract with Gulls' chairman Mike Bateson.
Later that month he became one of Neville Southall's first signings at Conference strugglers Dover Athletic, scoring the winner on his debut away to local rivals Margate on 26 December.