As an eight-year-old boy, a key experience strengthened his enthusiasm and he joined a youth theatre group at age eleven, taking over organisational duties some years later.

[8] Kaye's roles in recent international productions include drug baron Polo Yakur in Assassination Games (2011), the Duke of Milan Ludovico Sforza in The Borgias (2011-2013), Joshua Collins, father of the main character Barnabas Collins (Johnny Depp) in Dark Shadows (2012), famous Viking warrior Ivar the Boneless in Hammer of the Gods (2013), pub landlord Mick Sturrock in BBC One's crime show The Coroner (2015-2016) and the Saxon King Aelle of Northumbria in History channel's period drama Vikings (2013-2017).

[8] Aside from his villain roles as cold-blooded Polo Yakur (a "stand-out performance"[10]) and unscrupulous Duke Ludovico it is mainly his portrayal of King Aelle - in Vikings the deadly enemy and nemesis of the legendary hero of the show Ragnar Lothbrok (Travis Fimmel) - that has recently shaped Kaye's international profile.

Earlier internationally known roles include Reuben Starkadder in the television film Cold Comfort Farm (1995) and Adonija in the German-Italian-French co-production Solomon (1997).

[8] Up to 2021, Kaye has performed in four short films[15][8][16][17] and, as a voice artist, he has lent his voice to characters in several video games like Warhammer 40,000: Fire Warrior (2003) and Star Wars: The Old Republic (2011)[8] as well as starring as a main character in comedy episodes and short films produced by his own and Douglas McFerran's production company Comedy Ink Productions.

[18][19] From October 2017 to January 2018 he returned to the stage in the role of Tom Kettle in Jez Butterworth's play The Ferryman at Gielgud Theatre in London's West End.

[20][8] In April and May 2018, Kaye came back to the small screen in the UK in his role as Pesca in the five part miniseries The Woman in White, a new BBC adaptation of Wilkie Collins' novel of the same name.

A sneak peek review of British independent feature film For Love or Money: An Unromantic Comedy (released in the US on 15 March 2019) has emphasised his outstanding performance as the father of the main female character.

[35][36][37] In September 2022, Kaye rejoined The Celebs to mark the 40th anniversary of the Michael Jackson classic album Thriller and raise money for Great Ormond Street Hospital, with a new rendition of the title track.