After graduation from Gunnery and Torpedo School in early 1914, he was assigned to the destroyers Yunagi and Amatsukaze during World War I.

From June 1918, he served as a foreign naval observer in the Netherlands, and was member of the Japanese delegation at the Versailles Treaty negotiations in 1920.

He returned to sea as captain of the cruiser Yura in June 1933, followed by the Aoba in November 1933, Yakumo in February 1934, and battleship Hyūga in September 1935.

From November 1936, Sugiyama was appointed Chief of Staff of the IJN 3rd Fleet, and was thus in a senior command position of Japanese naval forces in China during the early stages of the Second Sino-Japanese War.

Following the start of the Pacific War, on January 3, 1942, Sugiyama accepted the post of commander in chief of the IJN 3rd Expeditionary Fleet.