His brother Raymond VI received him badly, fearing having to share his father's inheritance, and went so far as to contest Baldwin's birth.
[1] In 1211, Raymond entrusted him with the castle of Montferrand to defend against the army of Simon de Montfort who led the Albigensian Crusade.
He then negotiated with Simon, whom he recognized from court, and obtained the surrender of the castle in exchange for the lives of the inhabitants and freedom for the defenders.
[2] However, when he returned to Toulouse, his brother violently reproached him for having lost the castle and Baldwin decided to join the crusader army.
Traditionally, he was thought to be the progenitor of the extant House of Toulouse-Lautrec, however, recent DNA research has found that they are paternally descendants of the ninth century Counts of Albi.