Caleb, who was nicknamed PlayStation by the Puerto Rico Islanders supporters for his tireless work rate, grew up playing youth soccer for the Capital Area Soccer League and was North Carolina High School player of the year in 1996 when he helped lead Sanderson High School to the North Carolina State title.

During this time, he was also a member of the U17 and U18 United States men's national soccer team where he played alongside Tim Howard, Steve Cherundolo, Carlos Bocanegra and Chris Albright.

In 2000, in the most exciting 13 seconds of college soccer, a play that involved a 50-yard pass, three headers and a volley, Caleb scored the golden goal to bring the Tar Heels the Atlantic Coast Conference tournament title.

In 2004, Caleb returned to the United States with the Richmond Kickers of USL First Division (at the time) but transferred midseason to Charleston Battery of the same league.

At the end of that season, Norkus took an opportunity to play in the Primera División de Chile with Unión Española and helped the team to the 2005 Apertura title.

During the summer of 1997, he won the MVP Trophy at the Voreppe Tournament in Paris, France, and was subsequently invited to a try-out with Anderlecht of Belgium's Premier League.