Its earliest charters of endowment are of the reign of Henry II, but undated; but as a prior of Caldwell witnessed a charter granted by Robert de Brus, 2nd Lord of Annandale to Harrold during the lifetime of Malcolm IV of Scotland (1153–65), it may be concluded that this house, like so many others in Bedfordshire, was founded early in the reign of Henry II or perhaps in that of Stephen.

The priory belonged at first to the Canons of the Holy Sepulchre and was dedicated to St. John Baptist; but after the thirteenth century it probably ceased to be in any way distinguished from the other Augustinian houses.

[2] Four churches in Bedfordshire – Bromham, Roxton, Sandy and Oakley with the chapel of Clapham – belonged to Caldwell at the beginning of the thirteenth century; Marsworth and Broughton in Buckinghamshire, and Arnesby in Leicestershire before 1291; Tolleshunt Major in Essex at a later date.

[2] At the siege of Bedford Castle in 1224, the canons assisted the king by providing him with materials for mangonels, and received in return a share of the stones from the dismantled walls.

Robert of Houghton granted to the canons the site of the priory in 1272; and in 1336 they held lands and tenements in Bedford, Bromham, Milton, Colesden, Roxton, Chalverston, Sandy, Sutton, Potton, Thurleigh, Holwell, Felmersham and Shelton.

A part of the tithes from both of these had been granted to Osney Abbey at its foundation, amounting to a pension of 12 marks; and from the first the canons of Caldwell seem to have made efforts to escape this payment.

In 1279 they had to be ordered to pay it 'on pain of excommunication'; but in the beginning of the fourteenth century Hugh de Beauchamp, who was prior at the time, began a long series of suits with Osney on the same subject.

It was probably the pressure of poverty at this particular time that stirred the prior to make these efforts; he was then rebuilding the conventual church, and only a few years before Bishop Dalderby had granted a licence to the canons to beg alms for this purpose, as they were so poor.

The bishop wrote of it in the same year as 'a very poor place,' and said that instead of the £100 which the king had asked for in his letter, he had only instructed the prior to contribute £20 towards the loan which was being collected from all the religious houses.

There is nothing special in the injunctions of this time which might point to laxity; the bishop only said that the canons were not to go to Bedford, that hunting dogs were not to be kept in the monastery, and that the common seal was to be kept under lock and key.