[5][6] He was educated at St Joseph's College—the same school as his father—where he played junior football alongside future Geelong teammate Cameron Ling.

[7] Scarlett entered the next phase of his junior career when he was selected to play for the Geelong Falcons in the under 18 TAC Cup competition in 1997.

[8] His achievements with the Falcons were recognised when he was chosen to play for the Victorian Country representative side at the AFL Under 18 Championships,[8][9][10] as well as being selected in the full-back position of the TAC Cup's Team of the Year.

[10] In analysing his drafting, media reports identified Scarlett's strong marking and kicking skills whilst noting his potential as a key defender.

[10] Despite playing in the league's pre-season competition as a back pocket,[12] Scarlett did not make his senior-level debut until the final round of the 1998 AFL season where he was matched up against Essendon full-forward Matthew Lloyd.

[6] Geelong won this game by ten points, and although Scarlett himself garnered a respectable six disposals and took two marks, he allowed Lloyd to score six goals – a third of the team's total.

Scarlett later credited these match-ups with Lloyd as assisting his long-term development as a player – acknowledging that it was a challenging start to his career.

This capped off a breakthrough season for Scarlett, scoring two goals, achieving his first match in which he gathered at least ten possessions, and earning an AFL Rising Star nomination.

Scarlett shared his 100th match with fellow Geelong defender Tom Harley in a 23-point victory over Essendon in the ninth round of the season.

He gathered further individual accolades, finishing runner-up in club best and fairest voting, as well as earning an All-Australian selection for the second-consecutive season.

Scarlett's consistency, reflected with his appearance in twenty-five games during the year and ranking equal twelfth for total marks within the league (146), was rewarded with a runner-up finishing in the club best and fairest award.

Geelong progressed through to the 2007 AFL Grand Final, where they defeated Port Adelaide by a record 119 points to capture their first premiership since 1963.

Scarlett performed consistently, but failed to have a massive impact in the final, gathering 11 disposals and one mark, as well as laying two tackles.

Scarlett was vital in Geelong's 2009 Grand Final triumph where he produced some magic play with less than five minutes to go, kicking the stationary ball off the ground from the centre square and setting up the match and premiership-winning goal by Paul Chapman.

Scarlett married his longtime partner Milla Warren in October 2007,[20] which capped off his premiership-winning year and the addition of the couple's first child, a daughter Charlie, to the family in May.

[20] Scarlett co-owned a Geelong-based restaurant with fellow Geelong footballers Steven King and Kent Kingsley called the George and Dragon Hotel from 2004–2007 until it changed name and ownership due to financial difficulty.