The village growth was largely influenced by the needs of the Wrest Park estate and most of the inhabitants were servants, gardeners, stable hands and blacksmiths who lived in thatched and terrace cottages some of which still exist today.

Silsoe's position on a main road to London and halfway between Bedford and Luton made it a favourite halt for drovers and travellers stopping at the George coaching inn (first mentioned in 1624) on the High Street.

Silsoe had many inns including The White Hart, The Ragged Staff, Duke of Kent's Arms, The Bell, The Battle of Alma (or known as The Mouse's Hole in West End Road), Lord Nelson (Newbury Lane) and The George (High Street).

Lady Lucas, the last member of the de Grey dynasty, sold the House and estate before an auction that was to have been held at the Mart, Tokenhouse Yard, London EC2 on 17 July 1917.

Wrest Park's house and gardens are now part of English Heritage and have been restored and opened to the public from 4 August 2011.

In the early 19th century Thomas Philip Robinson, 2nd Earl de Grey (1781–1859), wished to construct a spire but the weight of the structure proved too much for the supporting walls and the whole building collapsed.

The architect was Smith of Hereford who achieved 'an antiquarian accuracy here extremely rare ten years before Pugin'.

Other sources such as Arthur Mee suggest that the architect was Earl de Grey, confirmed by Charles Read.

On Sunday, 25 July 1909 King Edward VII attended the church service here and made a donation towards the clock fund.

It was used as a temporary place for stray animals or drunks, and its central pole, now removed, was used to chain prisoners in transit between Bedford and Luton.

As an integral part of the village, Cranfield University had also been invited to contribute to Silsoe Parish Council's Millennium Time Capsule which was placed there in March 2000.

[11] The East Half House and Whitehall (Wrest Park) communities are in the catchment zone for Robert Bloomfield Academy.