[6] Rougvie signed for newly promoted English side Chelsea in 1984 for £150,000, a team which included the likes of Kerry Dixon, Pat Nevin and David Speedie.
Though initially a regular in the side, Rougvie's lack of pace and often reckless tackling were to hamper his progress with the club.
During a League Cup quarter-final match with Sheffield Wednesday at Hillsborough, Chelsea came from 0–3 down to lead 4–3, only for Rougvie to concede a last minute penalty with a careless challenge and cost the team victory.
He lost his place in the side towards the end of the 1984–85 season and thereafter only featured sporadically, though he did play in Chelsea's Full Members Cup win over Manchester City at Wembley; despite scoring an own goal, his team won 5–4.
[citation needed] He was sold to Brighton and Hove Albion in the summer of 1987 for £73,000 having made 100 appearances for Chelsea and scored three goals.