In 2006, he focused on completing his year twelve studies and playing football for Melbourne Grammar, where he was co-captain of the 1st XVIII.
His season was wrecked by injuries, but he managed to play several games for Hawthorn's VFL affiliate, the Box Hill Hawks.
Xavier played in the 2008 AFL Grand Final, where he amassed 28 disposals (the most for Hawthorn) and helped his team defeat Geelong.
In 2013, he was restricted to just two games after numerous calf injuries and at the end of the season he signed with the West Coast as an unrestricted free agent.
The following year, Ellis would replace Peter Bell for Channel Seven's coverage of AFL matches in Perth.