Karl was the eldest son of the Count Eitel Friedrich III of Hohenzollern (1494–1525) from his marriage to Johanna van Witthem (d. 1544), daughter of Philip, Lord of Beersel and Boutersem.
Karl was Imperial Archchamberlain and later chairman of the Aulic Council.
In 1534, he received the Counties of Sigmaringen and Veringen as imperial fiefs from Emperor Karl V. Karl married in 1537 with Anna (1512–1579), a daughter of Margrave Ernst of Baden-Durlach, with whom he had several children, among them: He had held the family possessions in a single hand since the Counts of Haigerloch had died out with the death of his cousin Jobst Nicholas II in 1558.
His eldest son, Eitel Friedrich IV became the founder of the Hohenzollern-Hechingen line.
The youngest son, Joachim, received the county of Zollern.