It was extended several times until the 17th century, with notable additions including a gatehouse, the western bastion, and the grand apartments.

Originally an oppidum, or defensive hill town, located at the southernmost border of the Norman duchy of William the Conqueror, Carrouges was vainly besieged by the Plantagenets in 1136.

Jean de Carrouges a vassal of Pierre II, Count of Alençon, became famous as one of the combatants in the last judicial duel to be permitted in France, in 1386.

These alliances gave Blosset the means of restoring and expanding the château, which had suffered great damage following its confiscation by Henry VI of England after the battle of Verneuil in 1424.

At the time of the French Wars of Religion (1562–98), the château was again strengthened, with the construction of the western bastion, but during the following century the sumptuous grand apartments were built.

In the 17th century, Tanneguy II Le Veneur, comte de Tillières (d.1652), was dispatched to England to negotiate the marriage of Henrietta Maria of France, sister of Louis XIII, to the future King Charles I. Tanneguy II lived on his estates at Tillières and left Carrouges to his brother Jacques, abbot of Silly.

At the end of the 18th century, Alexis Le Veneur, vicomte de Tillières (1746–1833), a soldier and progressive, was one of the supporters of the philosopher Jean-Jacques Rousseau.

The Grand Salon occupies one corner of the building, the straw-coloured woodwork dates from the end of the 17th and the beginning of the 18th century.