Caswell was born in Wednesbury and began his career with Walsall making his debut in the 1972–73 season.
[3] He became the regular in the "Saddlers" side whilst left back was the key position he was able to play in all outfield positions and was a key member of the side which lost narrowly to Liverpool in the semi-final of the League Cup in 1983–84.
[3] After racking up 400 league appearances for Walsall he joined Billy Bremner at Doncaster Rovers however in October 1985 Bremner joined Leeds United and the following month he signed Caswell for £30,000.
[3] He then joined Birmingham City's community team and then became youth coach.
[3] In 1992, he became youth coach at Stoke City until 1995 when he became assistant manager to Wayne Clarke at Telford United.