Originally known as Yestred (from the Brythonic Ystrad, meaning strath or dale), the barony of Yester was granted by King William the Lion to Hugo de Giffard, a Norman immigrant given land in East Lothian during the reign of King David I. Latin language documents note the location as Zester in vital records.

In May 1544 during the conflict known as the Rough Wooing, the castle, village, and harvest were burnt by the English army returning from the burning of Edinburgh.

In February 1548 the English commander Grey of Wilton captured the castle and put George Douglas of Pittendreich in charge of Yester and Dalkeith.

Two men suspected of shouting insults defaming Edward VI were made to fight a duel at the market place of Haddington.

The castle gradually fell into disrepair, and by the late 17th century was in a very parlous state, the stones having been much quarried for building material.

Old tales tell that his castle, or at least his cellar and keep, were wrought by witchcraft, for there is there a marvellous underground cavern wonderfully constructed and extending under a large area of ground.

A few hundred years later however, in 1692, on her wedding night, the fiancée of Sir George Broun, a Baronet of Nova Scotia and inheritor of the Colstoun estate, decided to remove the pear from its casket.

Robert with his two sons were soon after killed, en route to Edinburgh; they were swept away by a flash flood caused by the River Tyne bursting its banks.

[9] For his supposed role in the struggles between King Haakon of Norway and King Alexander, ultimately culminating in the Battle of Largs, Sir Walter Scott immortalises Giffard in Marmion: A Clerk could tell what years have flown Since Alexander fill’d our throne, (Third monarch of that warlike name,) And eke the time when here he came To seek Sir Hugo, then our lord: A braver never drew a sword; A wiser never, at the hour Of midnight, spoke the word of power: The same, whom ancient records call The founder of the Goblin-Hall.