His first choice in sports was basketball but soon he joined the most popular pre-war Latvian football club - Rīgas FK.
At that time RFK had a very strong squad which included Latvia national football team players Ēriks Pētersons, Sergejs Maģers, Fricis Kaņeps and others.
With Rīgas FK Ulbergs stayed until 1942, then in order not to be called into German army, he took up work on the railroad and joined the newly founded football club Daugavieši.
In 1944 Ulbergs was arrested because of a remark that one of his friends had made in a restaurant during Ēriks Pētersons birthday party about Adolf Hitler.
After the war Ulbergs joined the FK Dinamo Rīga but because of an injury he soon switched from playing to coaching.