Originally it had immense gardens below it, on the banks of the river, with fountains, stately homes and seven chapels, of which one (Cristo) remains.

A 17th-century work, building started in 1601 under commission from Francisco Gómez de Sandoval y Rojas, Duke of Lerma.

The columns are made of single pieces of granite (the quarries were owned by the duke).

On the upper level the columns do not support arches, but rather a running frieze with iron balconies between them, and windows which were originally made from walnut wood.

No palace could have more than two towers except those belonging to kings, but the Duke of Lerma was given this privilege due to his great powers in Court.