The medieval College had been founded on the earlier site of a Benedictine priory, originally located in Clare Castle, but moved to Stoke-by-Clare in 1124.

Under the patronage of the powerful de Clare family, it was one of the wealthiest monastic houses in Norman England, until a disastrous fire in the 1390s.

[1][2] In 1534 Dr Walter Haddon, writing in a letter from Cambridge, says of the college "how that place seemed in a manner to be made on purpose for scholars, both to learn themselves, and to teach others: and that its situation was such that above all others it is best suited for honest and ingenious pleasures."

The buildings had been abandoned after the Dissolution of the monasteries in the 1540s; the site was bought by the Elwes family around 1660, who created the surviving main house and stables.

The Elweses did not always keep the premises in fine style – one member of the family, John Elwes was so mean that he served as Dickens's model for Ebenezer Scrooge in A Christmas Carol.