Piccadilly was little more than a country lane, but the land to the north of it was just beginning to be used for housing; the next several decades would see the development of the whole of this area, which was to become London's leading aristocratic residential district, Mayfair.
Ironically in view of later events he always maintained that he had been reluctant to build such an ostentatious house, but was unable to rent any suitable mansion.
The house was built on the double pile plan, meaning that it was two rooms deep, and had two main storeys of roughly equal height.
Among the many allegations against him it was charged that he had appropriated, to build his house, stone intended for repairs to St. Paul's Cathedral after the Great Fire of London.
That same year, on 14 June 1667, Samuel Pepys recorded in his diary: "...some rude people have been... at my Lord Clarendon's where they cut down the trees before his house and broke his windows."
The building of the house and the resentment it caused are major elements in The Piccadilly Plot, the seventh of the Thomas Chaloner series of mystery novels by Susanna Gregory.