Born in Montreal, he was part of an ethnic French family that had been fur trading in New France since the late 17th century.
After completing formal education at the Jesuit college at Quebec, he joined his brothers Jacques and Antoine in a fur trading business they called Baby Frères.
Joining the governing class meant that Baby had less time to spend on commercial activities.
His son Charles-François-Xavier later served as a member of the Legislative Council of the Province of Canada.
His daughter Monique-Ursule married Thomas Ainslie Young, a member of the legislative assembly.