It was built with the features of a fortress during the period when the Catholic Church was at war with the a heretical sect called the Cathars, which originated in Albi.
Bishop de Combret, the next resident, further fortified his palace by connecting it to the Cathedral tower, twenty-five meters away.
He added bastions and a machicoulis over the entrance, to drop heavy objects or boiling water on any attackers.
In 1905, when the Cathedral and its properties were officially nationalised, and the Palace was given to the city of Albi for use as a museum.
In 1922 the museum received an important collection of works by Henri de Toulouse-Lautrec, donated by his mother.