A one-club man, Milne spent his entire professional career with Swansea Town where he holds the club record for most appearances in the Football League.

[2] After winning the Northumberland Senior Cup, Milne signed for Swansea Town in May 1920 from local junior side Walker Celtic for the price of a fish tea and a cup of tea (2/9d).

His final two appearances for Swansea came in the 1936–37 season as an emergency goalkeeper, replacing the injured Stan Moore for games against Leicester City and Nottingham Forest.

[2] Milne played a total of 586 league games while at Swansea, which is a record to this day, while only two other players have managed to top 500.

Roger Freestone made more appearances in all competitions for the Swans, though Freestone played in a number of competitions that did not exist in Milne's day, notably the Football League Cup, Football League Trophy, European Cup Winners' Cup and the Football League play-offs.